[GSoC] RTEMS Tester Improvements

Sun Mar 26 17:34:48 UTC 2017

Hello Chris, all,

Thank you for the explanatory answers.

I have updated my proposal as linked in the GSoC
wiki(https://devel.rtems.org/wiki/GSoC/2017). I'll be grateful if you
and the community members can view it and provide your valuable
feedback.

The portion where adding simulator support is discussed might look a
bit under-researched. I am working on it. Also, I am bit confused
whether to add coverity issues into the proposal. I still have to look
into it. Furthermore,  I would like to know if the work mentioned in
the proposal is enough work for the summer.

Also, do you suggest against using PyYAML to parse .yml configuration
files (since that will add a dependency into rtems-tester)?

Thank you,
Tanu Hari Dixit.

On Wed, Mar 22, 2017 at 10:54 AM, Chris Johns <chrisj at rtems.org> wrote:
> On 22/03/2017 03:46, Tanu Hari Dixit wrote:
>>
>> Thank you so much Chris for the elaborate answers.  I have a clearer
>> idea of the project now. Thank you Gedare for the tip, it definitely
>> comes in handy.
>> I'm sorry for the late reply. I have more questions and I will be
>> grateful if the devs can help me with these.
>
>
> Just to clarify something, the YAML change is for the .mc files and not the
> .cfg files. The configuration is to stay as is using the config module in
> the toolkit.
>
>> How does one use the console?
>
>
> The console refers to the RTEMS console output and the tester needs to
> support the various ways we can access a console. We currently have stdio
> (stdout/stdin) and termios on Unix. The task is to allow more consoles and
> to have Windows supported.
>
>> I see extensive use of termios in
>> rtems-tools.git/tester/rt/stty.py which is in turn used in
>> tester/rt/console.py
>
>
> The tty class is based on termios and this is the issue on Windows. I would
> leave stty.py as is and add a new serial.py file and class which is based on
> PySerial. The console module is the collection of all console interface
> methods the tester supports. You can see the stdio and tty classes, they are
> a kind of console class. Serial would added as a kinbd of console as well.
> The base console class defines the interface.
>
>> Can you please demonstrate one example so that I
>> get an idea?
>
>
> The stdio and tty. The configuration file parser ends up here on a
> `%console` directive (see console.cfg):
>
> https://git.rtems.org/rtems-tools/tree/tester/rt/config.py#n87
>
>> I believe PySerial will need to be configured in a
>> similar way i.e. making objects of class tty and assigning them
>> attributes with Pyserial instead of termios (like ignbrk and the
>> rest). Please correct me if I am wrong.
>
>
> I have not looked at the detail. It would follow a similar pattern to tty
> and tty_defaults:
>
> https://git.rtems.org/rtems-tools/tree/tester/rtems/testing/console.cfg#n47
>
>> Moreover, about the telnet console, I went through the ser2net
>> package. It serves as a good way to convert (or maybe the right word
>> is "use") serial connections via telnet.
>
>
> It is very useful.
>
>> I can make a simple bash
>> script to install ser2net and then change its conf file to fix the
>> appropriate port numbers and banners(which I can take as entry from
>> user).
>
>
> The set up and installation of ser2net is outside the scope of the work and
> we should not step into that topic. There are too many hosts and
> differences, for example I do not have bash installed.
>
>> Then I shall use them via telnetlib to communicate to the hosts
>> and then I can run tests via the console on the target. Is this the
>> right direction or should I be thinking differently?
>
>
> Yes, telnetlib would be another console. We would wrap telnetlib in a module
> and then add it to the console module.
>
>> Also I am really confused as to how rtemstoolkit comes into the
>> picture. Can you please provide me with a rough workflow of the
>> console?
>
>
> The tool kit provides a common based of functionality that can be used
> across a range of applications. The toolkit currently contains C++ and
> Python code. Code in the rtemstoolkit is not specific to any application,
> for example you could argue stty.py could be moved into the tool kit. We
> would do this later.
>
>> I also had questions regarding extending support for "expected-fail"
>> and "user-input" state as mentioned here
>> (https://devel.rtems.org/ticket/2946). What follows is my
>> understanding of what I need to do to extend support for
>> "user-input"(and "expected-fail"), please correct me if I am wrong.
>
>
> Expected fail is already done and user-input is something I will add. You do
> not need to be concerned about this ticket.
>
>> I
>> need to make a directive like DTEST_STATE_USER_INPUT and declare it to
>> be 1 in /rtems/src/rtems/tools/build/rtems-test-check and choose a
>> suitable TEST_STATE_STRING in buffer_test_io.h.
>
>
> It is -D as in a command line define to the compiler.
>
>> Consequently, parse
>> for it in tester/rt/report.py. (probably this has to be done for
>> "expected-fail" state also).
>
>
> We should track all states.
>
>> Also, I would need to extend the
>> attributes of class report that already has
>>         self.passed = 0
>>         self.failed = 0
>>         self.timeouts = 0
>>         self.invalids = 0
>> to also include self.expected_fails and self.user_inputs. Please
>> correct me if I am wrong.
>
>
> Yes. The GSoC project also needs to consider how we export per BSP
> information of the expected test results so we can compute any regressions,
> ie self.failed > bsp.failed is a regression if we had bsp values in the
> tester.
>
> Note, the report module is threaded so there are locks.
>
>> Can you please point me to the commit with
>> which you added support for "extended-fail"?
>
>
> https://git.rtems.org/rtems/commit/?id=28fda6279b82c07a673a8ebf875b77bb69695f1a
>
>> Furthermore, I was able to run the hello world example on
>> arm/xilinx_zync_a9_qemu with rtems-test as you demonstrated. Thank
>> you.
>
>
> Nice.
>
>> Also, I understand why you emphasize on keeping a plotting tool
>> different from rtems-tester.
>
>
> Great.
>
>> An aside question: Do you use winpdb to debug the python scripts in
>> rtems-tester? Do you recommend some other tool?
>
>
> I had not seen winpdb before. It looks nice. I see a FreeBSD port so I may
> try it.
>
> I tend so build tracing into the code and enable or disable it and leave the
> trace in the code. This way the support is there if it is needed again.
>
> Chris