[Libre-soc-dev] insndb: further directions

[Dmitry Selyutin]

Whilst working on C decoder, I've come to the conclusion that I need
to provide a summary for future directions on insndb.
The recent comment from Andrey made this conclusion even more strong.
Quoting Deus Ex:
"We have grown, but there is still much to be done. Many that live in
darkness that must be shown the way. For it is the dawning of a new
day."

The order for some tasks is irrelevant. However, some of them depend
on the others.
This list is huge, so I'd be glad if you could join. Perhaps we could
discuss tasks on an individual basis, not in this thread.


1. C assembler (simple vanilla instructions)
A counterpart for C disassembler (pronounced decoder in 979). After
979, I think this task is manageable.
Prerequisites: Python and C knowledge; binutils knowledge.

2. Supporting instruction aliases
Even if we still haven't reached an agreement on aliases for new
instructions, the old ones do not intend to be gone. This task, like
the previous one, requires a custom format, this time depicting that
there's an alias A to instruction B and/or C, but the operands D, E, F
are converted to G, H, I or K, L, etc. This task is simpler, because
there's apparently no tree-like hierarchy. A subset of this task is
aliases for operands, e.g. CR-related: what do eq or lt mean. Perhaps
there are more of these.
As for vanilla instructions, this is mandatory: we won't be able to
implement the whole subset of vanilla OpenPower ISA for assembler and
disassembler C library.
Prerequisites: Python knowledge, OpenPower ISA book nearby.

3. Support aliases in libopid (C assembler/disassembler library,
OpenPower Instruction Database)
Prerequisites: task #2; all its prerequisites apply here as well.
Preferably to be done after task #1 as well.

4. SVP64 specs and field generation (docs + code)
By specs, I mean all specifiers and remapping logic: RM, predicates,
etc. Currently all of these are written manually; whenever we update
the docs, the code must be updated accordingly. Plus this code is
written manually in binutils. The solution to this issue is to treat
these in the same way as we treat anything else -- instructions, basic
operands, etc. We should provide a format which specifies the spec
name, the bit range, and, likely, the validators.
The files src/openpower/decoder/power_fields.py and
src/openpower/insndb/core.py should provide a good place to start;
Mapping class is the base starting point. If we can create the format
which leads to Mapping creation, we can drop all custom classes in
insndb/core.py. The most challenging part would be to incorporate the
validation and bind the actual fields to the specifiers (e.g. that
specifying upon assembly /vli spec brings some logic to a life). It's
a difficult task, mostly due to the need to incorporate parts of the
logic in the format. Plus specs need to have a tree-like layout, which
complicates the structure even more. We must find a good balance
between being simple and being expressive.
Prerequisites: a hell lot of a time, patience, "never surrender"
philosophy. If you're still here -- a damn good Python knowledge,
since the implementation must construct both classes and instances.

5. SVP64 support in libopid
Once we provide a solid ground for libopid, we can incorporate SVP64 support.
Prerequisites: absolutely task #4; perhaps tasks #2 and #3, if we
choose to support aliases in SVP64.

6. Decouple insndb and related files
We currently incorporate everything into openpower-isa. For projects
which intend to re-use libopid -- binutils, clang, whatever -- this
might be an issue. Even if the generation artifacts are to be
committed there, I'd like to make synchronization with these projects
simple. Anyone should be able to just clone binutils repository, clone
libopid and all relevant data (CSVs, fields.text, enums, etc.) and
build libopid in place without the need to establish the whole
LibreSOC infrastructure. This is just too difficult for someone not
involved with the project, and I suspect our code generation would be
banned unless it's easy to reproduce and debug libopid. This must be a
standalone code, which is reused by multiple users.
Likely we'll have to decouple the whole infrastructure related to the
code generation: insndb, fields, many enumerations and similar stuff.
Perhaps we'll even have to decouple more components, or at least make
them not depend on redundant bits: nmutil, print-based logging, etc.
Prerequisites: likely git submodules and a lot of time to study the
components hierarchy. Preferably to be done after the previous tasks:

6. Switch SVP64 code in binutils to libopid
Currently binutils duplicate the relevant details in their code base.
This code gets rotten too fast.
Prerequisites: C, binutils. Tasks #1 and #5 are a must (so are their
prerequisites). Task #3 is a soft requirement iff opid is used only
for SVP64 (likely the good choice for the first iteration). This can
be done before task #6, but ideally the upstream contribution should
be done after. I suspect that, unless we provide a simple way to
reproduce the libopid standalone build, binutils won't accept this
code.

7. Further directions are provided as investigations but are unlikely
to happen due to time constraints and low priorities:
7.1. Check opportunities for switching the whole binutils code to libopid.
7.2. Investigate clang asm/dis based on libopid.

Tasks at #7 are possible though unlikely directions. They involve a
massive amount of work, at the same time being non-critical for our
purposes.


I'm sorry this email became so long, but I want to have all these
thoughts recorded somewhere. I'd be grateful for your replies.
Again, perhaps each of the subtasks deserves its own thread. But at
this point it was easier to cross-reference them in a single email.

-- 
Best regards,
Dmitry Selyutin