[PATCH 00/13] Replace mongoose with civetweb.

[Chris Johns]

On 26/04/2016 22:22, Christian Mauderer wrote:
> Am 26.04.2016 um 04:51 schrieb Chris Johns:
>> On 26/04/2016 07:22, Joel Sherrill wrote:
>>> Agreed 100%!
>>>
>>> NTP has an RSB recipe and there are others for various libraries which
>>> should be good examples. Shouldn't take long to do the recipe. Just ask
>>> questions rather than getting stuck.
>>>
>>> On Apr 25, 2016 7:03 PM, "Gedare Bloom" <gedare at rtems.org
>>> <mailto:gedare at rtems.org>> wrote:
>>>
>>>      On Mon, Apr 25, 2016 at 11:25 AM, Joel Sherrill <joel at rtems.org
>>>      <mailto:joel at rtems.org>> wrote:
>>>       >
>>>       >
>>>       > On Mon, Apr 25, 2016 at 4:22 AM, Christian Mauderer
>>>       > <christian.mauderer at embedded-brains.de
>>>      <mailto:christian.mauderer at embedded-brains.de>> wrote:
>>>       >>
>>>       >> Essentially I agree that it would be nice to build civetweb as an
>>>       >> external library especially with the different network stacks in
>>>      mind.
>>>       >> But there are some points that keep me from doing it:
>>>       >>
>>>       >> 1. I have really no Idea what would be necessary to build it
>>> as an
>>>       >> upstream project using RSB. If that are only something like
>>>      three simple
>>>       >> steps, it should be possible to squeeze it in the little time
>>> budget
>>>       >> that is left for the project. If I first have to analyze and
>>>      change a
>>>       >> lot of RSB, it would be challenging.
>>>       >>
>>>       > FWIW I think this can be a secondary goal. Switching over to
>>> the real
>>>       > upstream project with proper licensing is important.
>>
>> I see both as equally important and once the switch is made we have an
>> on going liability in the project to keep the packaging maintained so
>> considering it a bit more I suppose the long term issue is actually more
>> important to me.
>
> There is still some work left before we can use the upstream project. I
> more or less took over most of the patches that we applied to mongoose.
> Most of them should just need some clean up and don't really add any
> maintenance effort for the civetweb maintainers.

Excellent. Thank you so much of doing this.

> I already contacted the civetweb author regarding two of the patches.
> One was a new one (civetweb now uses a timegm function that we don't
> have: https://github.com/civetweb/civetweb/pull/298).
>
> The second one was a more complex one that _could_ lead to maintenance
> effort: We (to be exact: I) added some paramters to mongoose to
> influence stack size and scheduling. The maintainer of civetweb wasn't
> really enthusiastic about the way of the integration. He would prefer
> another interface. I tried to invite the RTEMS mailinglist into the
> discussion but till now I didn't get any responses:
>    https://lists.rtems.org/pipermail/devel/2016-April/014621.html
>    https://groups.google.com/forum/#!topic/civetweb/_Mul9PxgpBE

I see. If there exists a risk for Windows, Linux or any platform related 
configuration parameter getting bad values, why not force the value, or 
ignore the user setting, or raise an error on those platforms a 
parameter setting does not work on. This model is consistent, easy, and 
allows a finer per platform control of all parameters. The documentation 
for configuration is in a single location.

I have been looking after a mongoose set up for a client someone else 
set up. There has now been a couple of sets of eyes looking at the 
configuration and making minor tweaks. If there was another call-back, 
or overloaded define interface being used we would never have found it.

It would be a shame to add a call-back interface because the 
configuration becomes split and integration becomes disjointed. Also 
these days I prefer weak symbols over call-backs. Overloading the define 
with a call-out is tricky and hidden and it leaks internal details as a 
published interface.

>
>>
>>>       >
>>>       > When we do this, we incur some burden of "retraining" users to
>>>      consider
>>>       > a webserver an external addon. But that is the direction I think
>>>      we want
>>>       > to head with httpd, telnetd, ftpd, etc. So it is a good long term
>>>      goal.
>>
>> Agreed.
>>
>>>       >
>>>       >>
>>>       >> Chris, I think you know the RSB best: What steps do you think
>>>      would be
>>>       >> necessary?
>>>       >>
>>>       >
>>>       > Chris...
>>>       >
>>
>> There are a number of examples that show how to do this. Ping me if you
>> need specific help. Gedare has highlighted NTP and there is also NetNSMP.
>>
>
> Civetweb seems to have multiple build systems. One is a most likely
> handwritten Makefile
> (https://github.com/civetweb/civetweb/blob/master/Makefile). Another
> would be cmake. Is there any experience, which build system would be
> easier to integrate? Do we have any other non-automake projects?

Either can be made to work. I have no problem with cmake but I have it 
installed for other work so I am ok. Using cmake would make it a 
dependency for the RSB.

If you can supply a simple set of shell commands we should be able to 
bake an RSB recipe.

Also with patches, I recently added a change to the RSB that allows us 
to use patches from mailing list archives. This means a post to a list 
can be referenced and we do not need to include patches in the 
rtems-tools repo.

>
>>>       >>
>>>       >> 2. Currently there is one test case for mghttpd
>>>      (libtests/mghttpd01).
>>>       >> This is one of the few tests that check some networking
>>> functions in
>>>       >> RTEMS.
>>
>> I would be concerned if the only way we can do network testing is to
>> consume a 3prd party package's source into RTEMS. :)
>
> It's not the only test that tests network functions in RTEMS. But as far
> as I know, there are not many. If we remove it, it will be one less.

I do not have a simple answer. I have a radical one, split the testsuite 
in 2, with the part in RTEMS specific to the internal and API parts of 
RTEMS, and a second repo that is the tests that depend on "other" parts, 
eg 3rd party libraries. Those tests could configure which tests are 
built depending on the libraries detected. The work flow would need to 
be fast and easy to ensure a speedy compile/test cycle.

I am going on a bit about this stuff, but I feel it is important. In 
time we will have Buildbot running and we will need to get this stuff 
integrated and my hope is Buildbot, developers and users all use the 
same work flow.

>
>>
>>> Further we should not loose the ability to test software
>>>      when it
>>>       >> is build with RSB.
>>
>> I agree.
>>
>>> How would we handle tests for software in RSB?
>>>       >>
>>>       >
>>>       > That's a Chris question. We should aim to have tests which run
>>> with
>>>       > either network stack for RSB built network services. This is a
>>> good
>>>       > requirement on the plan to moving to separately built network
>>> stacks
>>>       > and services.
>>
>> Network testing is an open topic as is vertically integrated packages.
>> The testing issues exists with libbsd, lwip and the in-tree stack once
>> we move out of the source tree. As we flesh out the details of
>> vertically integrating packages we need to figure out now to manage
>> testing.
>>
>> What tests does civetweb provide? If the test is important I would
>> assume it is important to all users. Can civetweb build the test?
>>
>
> Civetweb has a number of tests. But I think that most would not work
> with RTEMS without changing them. In RTEMS we currently have a
> relatively simple testcase that just checks some basic functions.

OK.

>
> I don't think that we would be able to run either of them with the
> current RSB. And I don't have enough time to integrate a way for testing
> into RSB within the current project. So moving civetweb out of RTEMS
> currently means that it is not tested any longer with an RTEMS system
> (or only if someone does it manually after changing something).

I do not think the RSB is the way forward for testing. It could build 
tests as a package but that would be the extent. We have rtems-test in 
the tools repo but it is need of further work.

I do not think we will find a suitable answer for testing in the short 
term. I suspect a second testsuite package may be the answer.

Chris