[Libre-soc-isa] [Bug 686] create Power ISA test API

[bugzilla-daemon at libre-soc.org]

https://bugs.libre-soc.org/show_bug.cgi?id=686

--- Comment #42 from Luke Kenneth Casson Leighton <lkcl at lkcl.net> ---
(In reply to Jacob Lifshay from comment #41)
> one thing to think about, if there is an expected state, where do we get the
> expected results? 

either by manual inspection or by running an equivalent
function in python that is known-good (and also verified).

there are about.... 100 such tests already, and many more
are needed.

in the case of the Power ISA Conformance Documents we *have*
to assume the values listed are "correct".

and be able to enter them.  an ExpectedState allows that to
be formally specified, where at the moment it is nonstandard
and a bit of a mess.

> I think it might be better to just run multiple
> simulators/VMs and compare their results instead of isolating one of them as
> the "canonical expected result".

the idea is to do both.

1) compare simulators against simulators.
2) compare simulators against "expected results".


> building a custom result generator just to get the "expected result" can be
> alternatively seen as building a custom simulator to be used just for that
> specific test case, so I think it would be better to acknowledge that that
> is the case and that it is just like the other simulators, so should also be
> a simulator.

one immediate counterexample which shows this line of reasoning to be invalid
is the
OpenPOWER Foundation Compliance Test documents.

we are *required* to extract the listed values and *required* to demonstrate
that they have been run.

the easiest way to do that is to put them into "ExpectedState".

additionally it is not a problem at all to have unit tests that check
simulators.
the unit tests do not have to be sophisticated, they definitely do not
need to be a full simulator.

have you looked at any of the test_caller*.py unit tests?

the SVP64 ones have been absolutely essential and serve a secondary purpose
of acting as both examples and documentation.

additionally if we find flaws it is very easy to report "Expected Results"
by.pointing at the unit test, which will hsve a class instance, "ExpectedState"

thus i count approximately four possibly five separate and distinct very good
reasons why we need an ExpectedState.

this is why i have been asking you to complete the BCD test in python
(and only python)

it is *not* a waste of time or waste of effort.

-- 
You are receiving this mail because:
You are on the CC list for the bug.