[Libre-soc-dev] UNFREEZE and results of Financial Audit of 2021-02-052 and 2021-08-071 (was: freeze on all RFPs until a review of Financial Accountability has been carried out)

Wed Sep 20 16:17:41 BST 2023

(bcc'd to Bob Goudriaan, cc'd to 2022-0E at nlnet.nl)

with thanks to Andrey for being the second for confirmation
of Financial Records, we conclude (and confirm) that:

*  there are no irregularities in the amounts / totals of
  either of these two Grants, in the total amounts for
  each Milestone.

* there are no discrepancies involving the requested/available
 amounts for 2021-08-071 (cavatools).

* there *are* however some *seeming* discrepancies involving the
 *requested/available* amounts for 2021-02-052, cryptoprimitives
 but that MANUAL records (double-checked against email records
 dating back to 2022, vs the NLnet RFP manual records "t-1",
 "t-5" etc) are CORRECT.

on this basis i am now happy that it is possible to carry out
the absolutely essential double-checking of RFPs when they
come in (i.e. making sure that the amount requested tallies with
the available amount on *NLnet's* system), and am therefore, thanks to
Andrey's help as a Second, happy to lift the
freeze on RFPs.

now for some more detail on what was found during this Audit,
and some explanation.

firstly: our primary objective especially given that Financial
Fraud attempts have been made against NLnet, with scammers
attempting to infiltrate themselves into projects (one such
individual contacted me approximately 3 years ago), is to ensure
that the Accounts are inviolate and meticulously accurate,
providing no opportunity for an EU Auditor to call us out
on our RFPs.

if that happens it is extremely serious for NLnet: I learned
that FundingBox (another EU 3rd Party Financial Service Provider)
would need *every single project* reviewed if even one cursory
review by an Auditor showed even one small discrepancy. the
amount of work involved for NLnet given that they have i think
over 250 active projects, is just enormous, and we therefore,
if we are to expect to apply for future Grants, need to make
absolutely certain that our Records are 100% accurate.

but there is something even more important: if an EU Auditor
were to have cursory questions, NLnet's team *must* have answers
available to them.  this is why they do such stringent
Due Diligence on the MOU Tasks.

(OPF ISA WG MOU review with Michiel was over *90 minutes*
of mine and his time, to go through every single task,
and that was after several days discussion and phone calls
with the rest of the team, *in addition* to weeks of actually
writing the Tasks in the first place).

with that as context underpinning the importance of getting this
right, let me now go over the details of the Financial Review.

cavatools grant
---------------

the cavatools Grant was easy to verify.  the NLnet
RFP system, being entirely the new "Automated" system and
with all tasks submitted so far having a corresponding JSON
entry, the validation took 2 minutes to cut/paste from NLnet's
online tables, combine with cut/paste task_db/report from
bugzilla, 1 minute for me to go quickly through the list and
i assume a similarly-short and straightforward amount of
time for Andrey.

total review time:

     well under 10 minutes.

going forward on this task, i have made a brief write-up noting
that if new sub-tasks under a Milestone are created they MUST
go on the MOU, and that requires some bureaucracy with NLnet to
keep the EU Auditors happy. this needs to be properly written
up by someone https://bugs.libre-soc.org/show_bug.cgi?id=1126#c15

there are some additional caveats, so please read that procedure
CAREFULLY. in particular DO NOT put in an RFP in future if there
does not exist an entry in NLnet's database. the complications
it causes i will go over in the detail of the review of the
other grant.

the cryptoprimitives grant.
--------------------------

this one is complex for everyone (NLnet, LibreSOC, admin
for both, and for the EU to Audit). hence why i had to call the
freeze.

total review time:

     WELL OVER TWO HOURS.

background:

this Grant *PRE-DATEs* NLnet's new RFP system. it is one of
the ones "caught in the middle" (mid-2022) where NLnet (and
all contributors) were transitioning from a "Manual Email"
submission system.

in addition to that we did not have a clear idea of what
crypto-primitives would be done, and so chose to create
a few "TOP LEVEL" Milestones, where sub-tasks (to be added
to the MoU) would be added later.

unfortunately, both Jacob and I had to submit RFPs:

1) using the OLD email system
2) BEFORE the sub-tasks had been added to the MoU
  (and approved, with associated MoU update signed by
   Bob. see https://bugs.libre-soc.org/show_bug.cgi?id=1126#c15)

this latter (2) was an experiment that, due to the overwhelming
complexity for everyone, is just NOT to be repeated.  even just
attempting to do the Financial Review Audit today took around
three hours to cover around eight to ten tasks that had seemingly
discrepancies: it is most fortunate that there were none, but it
was much harder than it should have been.

the reason is that even trying to find the task was extremely
difficult: i had to read the entirety of every single RFP,
searching through the "Results Comments", hunting for the
required bug sub-task number, then manually totalling up
the amounts after separating them out from other sub-tasks
under the same RFP.

this reminds me of the manual (emailed) RFPs which whilst
consistent (to a large extent, being in machine-readable form)
caused severe headaches for NLnet for the 2+ years they used
email.

the added complication was that the MANUAL records ("t-1" of
2022-12-13) were not quite accurate as to either whom or the
date, making identification particularly tricky.

(Bob, Michiel, can we please have this changed in 2022-02-052

from:   2022-12-13 € 1 300 t-1 Mr L.K.C. Leighton
to:     2022-07-21 € 1 300 t-1 Jacob Lifshay

i have raised this to keep track
https://bugs.libre-soc.org/show_bug.cgi?id=1172

)

what these "Manual" records are is a hang-over from the old
emailed RFP system.  there are approximately TWO HUNDRED such
entries from 2019 to mid-2022, fortunately only around 5 of them
are for the cryptoprimitives Grant.

how NLnet handled them is to have a *SEPARATE* record in the
RFP system, which is ONLY PARTIALLY integrated into the rest of
the RFP database.  consequently whilst the "Total Grant Amount"
matches up (EUR 50,000) the "Amount available for which an
RFP can be submitted under a given Milestone" does **NOT**
have these "manual" database entries subtracted automatically.

this was the source of the "seeming" discrepancy.

considerable time and effort was spent to research this old
system and two of us had to double-check that yes, the amounts
"missing" from available-to-pay did in fact match with our
bugzilla Reporting which indicated clearly that those amounts
had in fact been paid.

i also had to dig out copies of the old emailed RFPs (which are
only in *my* inbox, as well as on my laptop and also of course
distributed across the various people whom i laboriously
managed as Project Admin to have them submit those RFPs in the
right way).

bottom line here is that yes, no actual discrepancies were
found, but actually checking that was pretty much hell,
and *could have been avoided*.

we have had mistakes in the past over the years: people have
made edits without realising the consequences (placing our
ability to Verify NLnet RFPs and avoid having a full EU Audit
called down on NLnet's head because it is INDISTINGUISHABLE
FROM FRAUD)

however these (also innocuous, unintentional, genuine mistakes)
have been one-off, a single change, usually when both of us have
been on IRC, and i was able to correct that single edit
immediately (or ask the person to make it).  no harm done,
because it was a single change, one email comes in from
the bugtracker, easy to spot, no pressure.

this situation was entirely different.

*only two hours prior* i had just described to Andrey the
procedure that needed to be enacted: to *under no circumstances*
change the budgetary amounts as could result in an incorrect
RFP being submitted (ESPECIALLY given the situation with
the "Manual" records) and this is tantamount to fraud.

i had just explained that NLnet's Finance Director,
Bob Goudriaan, *has* to Authorize the change, which means
*consulting NLnet*, first.

if Andrey had only made *one* unauthorized (genuine, mistaken) change to
the MoU budget amounts, it would have been easy
to fix, easy to notice, easy to revert: absolutely no need
whatsoever to call a freeze and Audit. playful reminder,
don't do it again, but no harm done, despite serious consequences
if it had gone un-detected.

but it was the fact that an overwhelming number of changes
were made simultaneously, which I was completely unable to
track *and am directly responsible for tracking as the Project
Lead on the MOUs*, that led me to enact the freeze.

one change: no problem at all.

multiple simultaneous changes: Audit-Tracking and Accountability
is completely destroyed, and confidence and trust by NLnet in our
ability to handle such large sums of money (keeping the
very real threat of scammers at bay) can only realistically
be restored by doing a full Audit, because the tracking *in my
head* and confidence in the integrity of the database (in the
face of risk of an EU Audit) is completely undermined.

i was therefore underatandably pissed off at the extra workload
heaped upon me right when i am supposed to be recovering from
PTSD.

I trust that this helps everyone to understand the
insane level of complexity and historical detail behind
the Accounting of this project, in managing what is now
well over half a million Euros over five years.

If anyone else would like to take over these responsibilities
they are more than welcome, but I will remain the Project Lead
on the MOUs, as a "backstop" so that this situation if it
occurs again does not place us in jeapordy, and you will
need - UNPAID - to learn the FULL extent of the Accounting Procedures.

going forward, then, there are two things:

1. the cryptoprimitives grant fortunately ends July 1 2024
  so the "mix" (complexity) will be behind us.

2. Accounting Procedures - and following them -
  are absolutely essential to document. here is just one:
  https://bugs.libre-soc.org/show_bug.cgi?id=1126#c15

  more will be needed.

lastly, i can only say that i am sorry i had to put my foot
down like this.  with scammers actively trying in the past
to defraud NLnet, it's real serious that we be completely
above-board at all times.

i am poignantly aware that this is a stressful time for all of
us. i really wish it were otherwise. projects like this are
supposed to be fun, engaging and enjoyable. i have hidden
pretty much 100% of the Accounting Admin from you all, so
that it *can* be an enjoyable experience. unfortunately as you
are keenly aware i cannot handle that heavy workload any longer,
but all of you *have* to learn and appreciate what it means,
what is involved, as we go forwards, *at the same time*
preserving our reputation as a trustable recipient of such very
large sums of money.

l.

On Wed, Sep 20, 2023 at 9:16 AM Luke Kenneth Casson Leighton <lkcl at lkcl.net>
wrote:
>
> https://bugs.libre-soc.org/show_bug.cgi?id=1170#c4
>
> i am bcc'ing both NLnet and Bob Goudriaan on this message.
>
> some unauthorized changes were made to the Financial Accounting
> records yesterday night, in direct violation of my clear
> instructions only a couple of hours before they were made, that
> under no circumstances were unauthorized changes to be made.
>
> this means that any RFP placed can no longer be 100% signed
> off (by me) as the Project Financial Administrator, as there
> could be a financial accounting irregularity requesting more
> money than has been Authorized by the European Union.
>
> the two Grants affected are:
>
> * 2021-02-052  https://libre-soc.org/nlnet_2021_crypto_router/
> * 2021-08-071  https://libre-soc.org/nlnet_2021_3mdeb_cavatools/

-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68