[Libre-soc-bugs] [Bug 422] Migrate away from nmigen Record

[bugzilla-daemon at libre-soc.org]

https://bugs.libre-soc.org/show_bug.cgi?id=422

Luke Kenneth Casson Leighton <lkcl at lkcl.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|CONFIRMED                   |DEFERRED

--- Comment #4 from Luke Kenneth Casson Leighton <lkcl at lkcl.net> ---
(In reply to whitequark from comment #3)
> > awygle's comment here that Record.connect is "not used" is incorrect
> 
> Which comment? I do not see anything that says "not used" on that page.

it's here:

https://github.com/nmigen/nmigen/pull/368

 awygle commented on Apr 24

Deprecate the unfortunate Record.connect method, and the Direction
functionality of Record as well


> > in the 75,000 lines of code in libre-soc, we make substantial use of
> > Record (the entire codebase relies on it), and also use Record.connect
> > in a number of places.
> 
> First: the amount of code relying on a bad design does not affect the fact
> that a bad design should be replaced with an improved one. It only affects
> the amount of effort that goes into providing compatibility with existing
> code, and into making migration easier.

i hear you: it's just that... hearing for the first time that Record is
going away is quite alarming.


> Second: you, personally, have been vocally opposed to the current design of
> the Record and asked for several aspects of it to be changed. The future
> deprecation and replacement will serve your needs as well, and has been
> planned, in part, with your project in mind.

that's really appreciated.  it's perhaps unfortunate that i am, because of
significant heavy focus on other parts of the project, just hearing about
the intended deprecation 

> 
> Third: if time constraints on your project prevent you from migrating off
> the existing Record design, you will have:
>   a) The deprecation grace period between, at least, release 0.4 (which is
> the earliest release where Record may be deprecated) and release 0.5 (which
> is the earliest where Record may be removed). The next release is 0.3.
>   b) the ability, after Record is removed upstream, to take the removed code
> and use it, unmodified other than for the imports, in your codebase
> indefinitely. This is another aspect that has been planned while considering
> your project.

interesting - and much appreciated.  i had absolutely no idea that this was
the case, that you'd been this thoughtful in the planning.  depending on
timing (0.5 pre- or post- Oct 2020) we therefore have a couple of
non-disruptive routes.

so i think we're good.  thank you for these details.  it allows us to
plan effectively.

in future, how can we ensure better communication, here?


> In my view, this means that you are not materially constrained in any way
> whatsoever by the upstream decisions on Record.

ok.  that's really good to know, and a huge relief.  i was very concerned
for a while.


> > any modifications to the IEEE754 FP library (45,000 lines of code)-
> > which critically depends on Record existing - come with a penalty of
> > requiring a *week* of 100% CPU on high-performance machines to run
> > several hundreds of thousands of unit tests and regression tests.
> 
> The run-time of your test suite should become significantly lower once you
> migrate to the release 0.3. This has been planned while considering your
> project (in fact you paid me for it) and discussed with your team members.

... bear in mind that Yehowshua is sufficiently busy that he simply does
not have time to keep me informed.

if you have had private conversations with him, then the majority of those
private discussions will in fact remain private, and you should take that
in to consideration.

also, Yehowshua is not the Technical Lead: i am the Technical Lead of the
Libre-SOC project - not Yehowshua (see more below), and Yehowshua is the CEO
(not the CTO) of a Corporation that "happens to be using Libre-SOC source
code".

bottom line is that if it's not been sent to or discussed on libre-soc.org
resources (which are in absolutely no way owned or controlled by the
USA-based Corporation known as "Systemes Libre"), then you should assume
by default that it has not reached the attention of the team.

given that we (Libre-SOC) are funded by NLNet for full transparency,
private technical conversations involving the Libre-SOC codebase and
matters relating to it are a strong "red flag" anyway.


> > i also feel compelled to point out that it is of some significant
> > concern to hear of major design decisions to remove critical functionality
> > for the first time via 3rd hand notifications where none of our team have
> > been involved or consulted as to the implications
> 
> I have informed the CEO of your corporation,

hang on, hang on - you're confusing the Corporation (which is yet to
be established) with Libre-SOC.  the two are completely separate entities
(a Charitable Foundation is in the process of being established for
Libre-SOC which will help make that clear).

i am directly responsible for the Libre-SOC codebase and its technical
direction.

Yehowshua is responsible for a commercial entity that *uses* Libre-SOC,
whose sole purpose and existence is to hold the Mask Rights for a taped-out
processor, with zero ownership of copyrighted code so that if it is attacked 
with back-to-back patent lawsuits, those attacks have zero impact on the
technical and strategic direction of Libre-SOC.

Yehowshua is therefore not involved in the actual technical decisions
because he is simply too busy with other matters [Masters Thesis].


> Immanuel Yehowshua, in an email
> dated 2020-06-21 that nMigen will hold bi-monthly IRC meetings that
> specifically aim to coordinate with downstream projects and discuss any
> concerns that may arise, and suggested he invite team members to attend.

for technical matters related to Libre-SOC, the best person to raise that
with is me, directly, and the best place to raise it is the mailing list,
due to the transparency requirements (i will review the contents and
discussion and raise a bugreport to keep track of its impact).

(Yehowshua did in fact send a message to the list about the meetings).

to make it clear: for matters related to the creation and commercial sale
of ASICs paid for by Systemes Libre investors and customers, it would be
appropriate to contact Yehowshua about that.


> None attended.

i simply missed it: i watch the IRC logs, and read them - i haven't had time
yet to let you know.  it was an interesting discussion.

i have to say that when reading the messages, i didn't recall seeing anything
about Record being removed.  clearly, if i had, given the huge impact, i
would have said something.


> In my view, I have sufficiently reached out towards your team.

to be completely open and honest with you i feel quite intimidated and not
very welcome.  i did not feel comfortable in directly participating, so
chose to review the logs after the meeting had taken place.  i meant to
let you know, i apologise.


> At the moment, Andrew Wygle has no formal authority in the project (he does
> not even have a commit bit), and he made this pull request specifically and
> only because I asked him to work on it.

ok. appreciated.

> > also it would be disruptive to Minerva, where Record.connect is utilised
> > for all wishbone bus communications (and we are using that code)
> 
> A Minerva stakeholder is aware of the change, has attended the last meeting,
> and expressed no particular concern about it.

the Minerva codebase is only 4,000 lines.  Libre-SOC's codebase is literally
20 times larger, and it will likely be 40 to 50 times larger by the time
we have full functionality.

i would therefore expect that it be fairly easy for Minerva to migrate,
which would explain why the Minerva team has no issue with the plans.


> > it would be *significantly* disruptive to the project to be forced to
> > modify and review 75,000 lines of code in order to remove Record, and
> > it would be seriously problematic if Record.connect disappeared.
> 
> I must remind you that you are using a library versioned in the 0.x series
> that says, in the very first sentence of README, that it is "incomplete and
> under active development". 

i do appreciate that.  really :)

> In spite of that, I went above and beyond what is
> typically done in open-source projects (much less young open-source
> projects) to provide extensive migration assistance for user designs at
> every point in the past.

although you've not spelled it out, i could tell even from the initial
discussions back 2 years ago that you had this kind of appreciation and
understanding of APIs and the burden it places [on you] for maintaining
them.  although i do recall that you were quite stressed about doing so :)

without actually discussing it explicitly, i completely "got" that, with
your valuable committment to principles that were unspoken, nmigen would be
a much stronger base for us to create such a hugely ambitious commercial
ASIC on than any other HDL out there, even as nmigen progressed and
developed.

so i do "get it" - i just freaked out for a minute, given the time-pressure
and the attention that's on this project to succeed [and consequently,
logically, will also be on nmigen.  sorry about that!].


> In my view, your lack of effective planning and communication

whilst understandable i have to say that this is not very fair.
we're under huge time-constraint pressure, whitequark, and it's
a massive project.  for the Oct 2020 deadline i have over 30 *major*
tasks to keep track of... *at the first level*.

i'm barely keeping up with development, let alone external communication.
normally for a project of this size and scope there would be a full-time
Project Manager dedicated exclusively to dealing with task management and
team coordination, and that role is - because there is nobody else to do
it - my responsibility *as well* as being the key active technical developer.


> does not constitute an emergency on my side.

i agree, and on learning that you've thought significantly about a
route for us to continue to use Record (externally if necessary) is
greatly appreciated.  it means that we can plan a migration on a
suitable timescale rather than having to drop everything and scramble,
right at a critical juncture.

i won't close this bugreport, because we need something to plan the
migrations (after Oct 2020).  marking this as "Deferred" though.

thank you for taking the time to keep us informed, whitequark.

-- 
You are receiving this mail because:
You are on the CC list for the bug.