[GSoC] RTEMS Tester Improvements

[Chris Johns]

On 22/03/2017 03:46, Tanu Hari Dixit wrote:
> Thank you so much Chris for the elaborate answers.  I have a clearer
> idea of the project now. Thank you Gedare for the tip, it definitely
> comes in handy.
> I'm sorry for the late reply. I have more questions and I will be
> grateful if the devs can help me with these.

Just to clarify something, the YAML change is for the .mc files and not 
the .cfg files. The configuration is to stay as is using the config 
module in the toolkit.

> How does one use the console?

The console refers to the RTEMS console output and the tester needs to 
support the various ways we can access a console. We currently have 
stdio (stdout/stdin) and termios on Unix. The task is to allow more 
consoles and to have Windows supported.

> I see extensive use of termios in
> rtems-tools.git/tester/rt/stty.py which is in turn used in
> tester/rt/console.py

The tty class is based on termios and this is the issue on Windows. I 
would leave stty.py as is and add a new serial.py file and class which 
is based on PySerial. The console module is the collection of all 
console interface methods the tester supports. You can see the stdio and 
tty classes, they are a kind of console class. Serial would added as a 
kinbd of console as well. The base console class defines the interface.

> Can you please demonstrate one example so that I
> get an idea?

The stdio and tty. The configuration file parser ends up here on a 
`%console` directive (see console.cfg):

https://git.rtems.org/rtems-tools/tree/tester/rt/config.py#n87

> I believe PySerial will need to be configured in a
> similar way i.e. making objects of class tty and assigning them
> attributes with Pyserial instead of termios (like ignbrk and the
> rest). Please correct me if I am wrong.

I have not looked at the detail. It would follow a similar pattern to 
tty and tty_defaults:

https://git.rtems.org/rtems-tools/tree/tester/rtems/testing/console.cfg#n47

> Moreover, about the telnet console, I went through the ser2net
> package. It serves as a good way to convert (or maybe the right word
> is "use") serial connections via telnet.

It is very useful.

> I can make a simple bash
> script to install ser2net and then change its conf file to fix the
> appropriate port numbers and banners(which I can take as entry from
> user).

The set up and installation of ser2net is outside the scope of the work 
and we should not step into that topic. There are too many hosts and 
differences, for example I do not have bash installed.

> Then I shall use them via telnetlib to communicate to the hosts
> and then I can run tests via the console on the target. Is this the
> right direction or should I be thinking differently?

Yes, telnetlib would be another console. We would wrap telnetlib in a 
module and then add it to the console module.

> Also I am really confused as to how rtemstoolkit comes into the
> picture. Can you please provide me with a rough workflow of the
> console?

The tool kit provides a common based of functionality that can be used 
across a range of applications. The toolkit currently contains C++ and 
Python code. Code in the rtemstoolkit is not specific to any 
application, for example you could argue stty.py could be moved into the 
tool kit. We would do this later.

> I also had questions regarding extending support for "expected-fail"
> and "user-input" state as mentioned here
> (https://devel.rtems.org/ticket/2946). What follows is my
> understanding of what I need to do to extend support for
> "user-input"(and "expected-fail"), please correct me if I am wrong.

Expected fail is already done and user-input is something I will add. 
You do not need to be concerned about this ticket.

> I
> need to make a directive like DTEST_STATE_USER_INPUT and declare it to
> be 1 in /rtems/src/rtems/tools/build/rtems-test-check and choose a
> suitable TEST_STATE_STRING in buffer_test_io.h.

It is -D as in a command line define to the compiler.

> Consequently, parse
> for it in tester/rt/report.py. (probably this has to be done for
> "expected-fail" state also).

We should track all states.

> Also, I would need to extend the
> attributes of class report that already has
>         self.passed = 0
>         self.failed = 0
>         self.timeouts = 0
>         self.invalids = 0
> to also include self.expected_fails and self.user_inputs. Please
> correct me if I am wrong.

Yes. The GSoC project also needs to consider how we export per BSP 
information of the expected test results so we can compute any 
regressions, ie self.failed > bsp.failed is a regression if we had bsp 
values in the tester.

Note, the report module is threaded so there are locks.

> Can you please point me to the commit with
> which you added support for "extended-fail"?

https://git.rtems.org/rtems/commit/?id=28fda6279b82c07a673a8ebf875b77bb69695f1a

> Furthermore, I was able to run the hello world example on
> arm/xilinx_zync_a9_qemu with rtems-test as you demonstrated. Thank
> you.

Nice.

> Also, I understand why you emphasize on keeping a plotting tool
> different from rtems-tester.

Great.

> An aside question: Do you use winpdb to debug the python scripts in
> rtems-tester? Do you recommend some other tool?

I had not seen winpdb before. It looks nice. I see a FreeBSD port so I 
may try it.

I tend so build tracing into the code and enable or disable it and leave 
the trace in the code. This way the support is there if it is needed again.

Chris