[GSoC] RTEMS Tester Improvements

Sun Mar 26 17:35:45 UTC 2017

Thank you, Gedare, for the comments.

Regards,
Tanu Hari Dixit.

On Sun, Mar 26, 2017 at 11:04 PM, Tanu Hari Dixit <tokencolour at gmail.com> wrote:
> Hello Chris, all,
>
> Thank you for the explanatory answers.
>
> I have updated my proposal as linked in the GSoC
> wiki(https://devel.rtems.org/wiki/GSoC/2017). I'll be grateful if you
> and the community members can view it and provide your valuable
> feedback.
>
> The portion where adding simulator support is discussed might look a
> bit under-researched. I am working on it. Also, I am bit confused
> whether to add coverity issues into the proposal. I still have to look
> into it. Furthermore,  I would like to know if the work mentioned in
> the proposal is enough work for the summer.
>
> Also, do you suggest against using PyYAML to parse .yml configuration
> files (since that will add a dependency into rtems-tester)?
>
> Thank you,
> Tanu Hari Dixit.
>
> On Wed, Mar 22, 2017 at 10:54 AM, Chris Johns <chrisj at rtems.org> wrote:
>> On 22/03/2017 03:46, Tanu Hari Dixit wrote:
>>>
>>> Thank you so much Chris for the elaborate answers.  I have a clearer
>>> idea of the project now. Thank you Gedare for the tip, it definitely
>>> comes in handy.
>>> I'm sorry for the late reply. I have more questions and I will be
>>> grateful if the devs can help me with these.
>>
>>
>> Just to clarify something, the YAML change is for the .mc files and not the
>> .cfg files. The configuration is to stay as is using the config module in
>> the toolkit.
>>
>>> How does one use the console?
>>
>>
>> The console refers to the RTEMS console output and the tester needs to
>> support the various ways we can access a console. We currently have stdio
>> (stdout/stdin) and termios on Unix. The task is to allow more consoles and
>> to have Windows supported.
>>
>>> I see extensive use of termios in
>>> rtems-tools.git/tester/rt/stty.py which is in turn used in
>>> tester/rt/console.py
>>
>>
>> The tty class is based on termios and this is the issue on Windows. I would
>> leave stty.py as is and add a new serial.py file and class which is based on
>> PySerial. The console module is the collection of all console interface
>> methods the tester supports. You can see the stdio and tty classes, they are
>> a kind of console class. Serial would added as a kinbd of console as well.
>> The base console class defines the interface.
>>
>>> Can you please demonstrate one example so that I
>>> get an idea?
>>
>>
>> The stdio and tty. The configuration file parser ends up here on a
>> `%console` directive (see console.cfg):
>>
>> https://git.rtems.org/rtems-tools/tree/tester/rt/config.py#n87
>>
>>> I believe PySerial will need to be configured in a
>>> similar way i.e. making objects of class tty and assigning them
>>> attributes with Pyserial instead of termios (like ignbrk and the
>>> rest). Please correct me if I am wrong.
>>
>>
>> I have not looked at the detail. It would follow a similar pattern to tty
>> and tty_defaults:
>>
>> https://git.rtems.org/rtems-tools/tree/tester/rtems/testing/console.cfg#n47
>>
>>> Moreover, about the telnet console, I went through the ser2net
>>> package. It serves as a good way to convert (or maybe the right word
>>> is "use") serial connections via telnet.
>>
>>
>> It is very useful.
>>
>>> I can make a simple bash
>>> script to install ser2net and then change its conf file to fix the
>>> appropriate port numbers and banners(which I can take as entry from
>>> user).
>>
>>
>> The set up and installation of ser2net is outside the scope of the work and
>> we should not step into that topic. There are too many hosts and
>> differences, for example I do not have bash installed.
>>
>>> Then I shall use them via telnetlib to communicate to the hosts
>>> and then I can run tests via the console on the target. Is this the
>>> right direction or should I be thinking differently?
>>
>>
>> Yes, telnetlib would be another console. We would wrap telnetlib in a module
>> and then add it to the console module.
>>
>>> Also I am really confused as to how rtemstoolkit comes into the
>>> picture. Can you please provide me with a rough workflow of the
>>> console?
>>
>>
>> The tool kit provides a common based of functionality that can be used
>> across a range of applications. The toolkit currently contains C++ and
>> Python code. Code in the rtemstoolkit is not specific to any application,
>> for example you could argue stty.py could be moved into the tool kit. We
>> would do this later.
>>
>>> I also had questions regarding extending support for "expected-fail"
>>> and "user-input" state as mentioned here
>>> (https://devel.rtems.org/ticket/2946). What follows is my
>>> understanding of what I need to do to extend support for
>>> "user-input"(and "expected-fail"), please correct me if I am wrong.
>>
>>
>> Expected fail is already done and user-input is something I will add. You do
>> not need to be concerned about this ticket.
>>
>>> I
>>> need to make a directive like DTEST_STATE_USER_INPUT and declare it to
>>> be 1 in /rtems/src/rtems/tools/build/rtems-test-check and choose a
>>> suitable TEST_STATE_STRING in buffer_test_io.h.
>>
>>
>> It is -D as in a command line define to the compiler.
>>
>>> Consequently, parse
>>> for it in tester/rt/report.py. (probably this has to be done for
>>> "expected-fail" state also).
>>
>>
>> We should track all states.
>>
>>> Also, I would need to extend the
>>> attributes of class report that already has
>>>         self.passed = 0
>>>         self.failed = 0
>>>         self.timeouts = 0
>>>         self.invalids = 0
>>> to also include self.expected_fails and self.user_inputs. Please
>>> correct me if I am wrong.
>>
>>
>> Yes. The GSoC project also needs to consider how we export per BSP
>> information of the expected test results so we can compute any regressions,
>> ie self.failed > bsp.failed is a regression if we had bsp values in the
>> tester.
>>
>> Note, the report module is threaded so there are locks.
>>
>>> Can you please point me to the commit with
>>> which you added support for "extended-fail"?
>>
>>
>> https://git.rtems.org/rtems/commit/?id=28fda6279b82c07a673a8ebf875b77bb69695f1a
>>
>>> Furthermore, I was able to run the hello world example on
>>> arm/xilinx_zync_a9_qemu with rtems-test as you demonstrated. Thank
>>> you.
>>
>>
>> Nice.
>>
>>> Also, I understand why you emphasize on keeping a plotting tool
>>> different from rtems-tester.
>>
>>
>> Great.
>>
>>> An aside question: Do you use winpdb to debug the python scripts in
>>> rtems-tester? Do you recommend some other tool?
>>
>>
>> I had not seen winpdb before. It looks nice. I see a FreeBSD port so I may
>> try it.
>>
>> I tend so build tracing into the code and enable or disable it and leave the
>> trace in the code. This way the support is there if it is needed again.
>>
>> Chris