[Libre-soc-dev] nmigen tutorials etc.

[Andrey Miroshnikov]

On 10/10/2021 14:20, Luke Kenneth Casson Leighton wrote:
>> What came out of my exploration is a new tutorial section:
>> https://libre-soc.org/docs/learning_nmigen/
> 
> nice - that's really useful https://libre-soc.org/docs/nmigen_verilog_tb.png
I thought that since I'm starting to use a lot of these tools as a 
beginner, may as well brush up the documentation, clear up any out of 
date info, update commands etc.

Once my skills improve I imagine time will become more constrained 
(given the big April deadline we need to hit), so now is the time for 
admin stuff while learning.

>> Have a look, see if it makes sense.
> 
> excellent, i cross-referenced on https://libre-soc.org/resources
Nice

>> Also some of the nmigen tutorial links have been taken down, so I
>> replaced them with equivalent
> 
> actually you can still get many of them by using archive.org
Great idea! I'll check to see what I replaced and find the equivalent 
archive.org links.

> the key one is - and now that you understand this is AST - is
> assigning a massive complex batch of nmigen AST to a *python*
> variable, then using that *python* variable multiple times.
> 
> guess what happens?
> 
> that AST gets replicated... once for each *and every* time you use
> that python variable.
> 
> whoops.
You mean fun right? And many sleepless nights? XD
I've been hand-writing Robert's tutorial (not word for word though), to 
make sure I get the concepts ingrained. Better take out all the stops 
when it comes to learning nMigen.

> jacob just wrote this which helps, there
> http://bugs.libre-soc.org/show_bug.cgi?id=722 but hoo-boy are the
> memory requirements massive.
I get the gist (remove redundant logic instantiations?), but there's a 
lot of python I'm seeing for the first time. At least now I know what 
function decorator is haha:
https://docs.python.org/3.7/glossary.html#term-decorator

At least with ASTs there are pruning techniques to remove redundancies 
(as with other tree types), right?

> the alternatives - which i describe in detail here - are absolutely
> flat-out bat-s*** insane.
> https://libre-soc.org/3d_gpu/architecture/dynamic_simd/
Gave it a read. Now I finally have an idea of what PartionedSignal is 
meant to do. With a signal example it made much more sense.
Parallel bit groups that can be operated on separately, or combined 
together if no partitions are enabled. For nMigen to automatically 
expand those if-statements is a no-brainer.

Other than the surface level, I'll need to continue reading other 
material, and re-read it a few times. There's a lot of computer science 
theory I'll be catching up on :)

What I can't think of is an example use-case? Can you give an example of 
where one might want to take a dataword, split it into partitions, and 
operate in tandem?

Pixels for a frame buffer?
Velocity calculations?
Hashing/encryption algos?

> ah.  right.  you've possibly fallen into the common trap of "SIMD === Vectors".
> given that multiple very large Corporations' Marketing Machines have
> peddled that for over a decade this is not a surprise.
> 
> i updated the Vector Processor page on wikipedia to correct this
> https://en.wikipedia.org/wiki/Vector_processor
> 
> so no: i am *not* talking specfically about vector instructions, because
> they are not synonymous.
> 
> we *happen* to have some Vector instructions (SVP64).
> 
> the architectural back-end *happens* to be implemented in (masked, predicated)
> SIMD.
This is still unclear to me, but it'll become clearer as I continue 
reading and studying. (a lot of bookmarked material/videos to go through)
If anything, better that I started clueless, hopefully won't have any 
mistaken lines of thought.

>> With my limited knowledge, it seems nmigen has more powerful
>> abstractions,
> 
> yes, exactly.  but it's more than that.  have a look at how PowerDecoder
> is done.  see how long it takes you to *actually* spot any *actual*
> nmigen code in it.
> https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/decoder/power_decoder.py;hb=HEAD
> 
Had a brief look at the code, nMigen did start pretty far down, most of 
the start is data structure setup.

> there's actually two tutorials about it, *and* it's the subject of a number
> of the different talks i've done.
Before I go further in that direction I should probably get better hold 
on nMigen first.

> you'll love this: i recovered the GardenSnake.py example from python-ply,
> added a 3rd lexer pass which allowed it to properly support full python
> syntax, then used it as the basis for the openpower ISA pseudo-code
> to python compiler.
Great use of public domain code ;)
One of my friends in recent years has been a bigger proponent of Python, 
while I was moving more towards low-level. However now that I've been 
exposed to its flexibility, it's starting to grow on me. At least for 
applications not constrained by time, making very complex behaviour is 
almost trivial (like extending a compiler).

>> Shame such a concept isn't discussed much on the hardware side of
>> things. We have Karnaugh maps, KCL/KVL, which you can use to construct
>> diagrams and expressions to optimise.
> 
> this is 90s-era. not that hardware *needs* to, and it's so fundamental
> that i learned about karnaugh maps as part of O-Level and A-Level
> maths back in 1985-87, but hardware engineering hasn't moved on
> since...
90s?...hahaha most of these concepts are at least 100 years old!

> ... oh except for proprietary companies creating tools that you can only
> use under NDA.
I understand paying for a tool when it saves you time and money. Most 
tools I've seen however, intend to keep the user busy, instead of using 
software eng. techniques for automation, as you have described. Digital 
domain (of all things) should be the most automatable.

> and it takes so long that they can't possibly do design-iteration.
I must admit, I've been having a paradigm shift now after watching the 
conversations here. Before my bias was mainly toward lower-level design. 
However when abstractions are well-known and predictable, I see no issue 
with utilising them as much as possible.

> yes but they then have to be trained in software engineering techniques,
> and many of them can't handle it, there's too much they don't know.
> (which explains the programmer shortage: people who *have* been
> trained in software engineering get snapped up)
Yep, opportunities too lucrative for the best programmers!

> IBM *only* employs people with c++ software engineering experience
C++ still terrifies me a little. Almost like Javascript and Java, C++'s 
abstract concepts have almost nothing to do with C, and with each 
release they seem to pile up more and more. Didn't the first C++ 
compilers take a really long time before they became viable?
> to work on the IBM POWER processor HDL. this tells you everything you
> need to know. their entire validation suite is in c++.  they have their
> *own gate-level simulator*... in c++.   they even have their own
> HDL-to-GDS-II synthesis tools because the ones produced by commercial
> companies are incapable of dealing with 18 billion transistor designs.
Now that actually sounds like progress XD



On 10/10/2021 14:26, lkcl wrote:
 > Andrey can i suggest dropping the list links into the page as well, 
for context.
 > at least this one:
 > 
http://lists.libre-soc.org/pipermail/libre-soc-dev/2021-October/003858.html
Not sure what you meant here.

Do you mean add the mail archive link to the tutorial page?


 > and one about the reason why to use "show top"
And as for "show top" do you want an explanation ("block diagram is 
useful for visualising the internal architecture and compare with 
code...etc.")?


Andrey