[PATCH 00/13] Replace mongoose with civetweb.

[Chris Johns]

On 29/04/2016 16:19, Christian Mauderer wrote:
> Am 29.04.2016 um 02:45 schrieb Chris Johns:
>> On 28/04/2016 18:27, Christian Mauderer wrote:
>>> Am 27.04.2016 um 02:07 schrieb Chris Johns:
>>>> On 26/04/2016 22:22, Christian Mauderer wrote:
>>>>> Am 26.04.2016 um 04:51 schrieb Chris Johns:
>>>>>> On 26/04/2016 07:22, Joel Sherrill wrote:
>>>>>>>
>>> I put some work into building civetweb using RSB. My configuration is
>>> now able to build a "libcivetweb.a" using the Makefile of civetweb.
>>
>> Excellent. GNU make is available in the RSB on all hosts.
>>
>> How do we package this Makefile?
>
> The Makefile is the one provided by the civetweb repository. So we
> shouldn't pack it ourselves.

Oh sorry, I misunderstood and thought you had written one.

>
>>
>> We have a few things happening we need to land in a coordinated manner.
>> We have this work, libbsd, LwIP, and the removal of the current stack
>> from the RTEMS source tree. If we do not attempt to coordinate these
>> pieces work will be repeated, code duplicated and users will be given a
>> confusing inconsistent view of networking.
>>
>> What you have done here is really important and you are breaking new
>> ground. We also have a GSoC project under-way for LwIP and Pavel and I
>> have been discussing that work. There are a number of strikingly similar
>> issues appearing.
>>
>> We are currently thinking of a repo for LwIP on git.rtems.org, eg
>> rtems-lwip.git. It will contain some build system support files,
>> license, a readme (I suppose) and a submodule reference to the upstream
>> LwIP repo. The submodule would be a specific hash of the upstream
>> project. At release time the release process creates a single tarball of
>> all source, I do this for libbsd now.
>>
>> Do you think this would work for civetweb?
>
> I'm not sure if this would be the right way. This sounds more like
> forking civetweb. This seems overkill for only a few patches that
> hopefully can go away in the future.

I misunderstood and thought you had build system files.

> If we want to keep a own repository
> up to date with the upstream repository this means an ongoing effort.

I had a long chat to Amar about this topic today and he told me we 
should follow the FreeBSD model for vendor software. FreeBSD has 
https://svnweb.freebsd.org/base/vendor/ and it contains the vendor 
source untouched and then the version used is a copy with local changes. 
If you follow the link the bind directory is the untouched vendor source 
and bind9 is a copy of the bind v9 vendor source with the local changes. 
His view is we create rtems-vendor.git and bring all the code into this 
repo. He said we should bring all the networking stacks and libbsd each 
as a separate directory. He said it brings into a single place the 
history of all the packages we have to manage and it lets us combine 
this code with the tests. It means users have a single place to go for 
any vendor source. It is a change in direction from the RSB per package 
approach but if we adopt rtems-vendor.git I do not mind. It also allows 
the testing framework he has been working to be used. I will let him 
explain this more as it is not area I know too much about.

>
>>
>> What is not clear to me is do we following the LwIP path and a separate
>> repo for this package or do we have an 'rtems-net-services.git' repo
>> this code is part of? A combined repo would mean the build system used
>> is either a mix based on each part or we unify it to a common build
>> system. In time we will have other pieces of code added, eg telnetd etc
>> as we moved that from the in source tree.
>>
>> I see Sebastian removed the unreferenced services directory from libbsd
>> this week (the vc email was too big for the list) and thank you
>> Sebastian. It did make me wonder how we manage these things.
>>
>> We also need to unify the network configuration for testing. Currently
>> libbsd is inconsistent. Some tests use the waf configure command line
>> settings and others do not and force DHCP. I am looking to change this
>> so we can have a better solution that allows more complex
>> configurations. The intention is this can be used for LwIP as well. I
>> will post my ideas when I get time.
>>
>>> The next steps would be to remove mghttpd from RTEMS. Currently that
>>> means that I have to remove the test too. Any ideas where we could move
>>> it to? If we don't have a solution: Do we really want to leave the code
>>> untested until someone has the time and budget to implement a external
>>> test suit?
>>
>> This is a really good question. I do not know and it also relates to how
>> we package this code.
>
> This is a question at hand. So I don't think that we can avoid it if we
> want to remove mongoose from RTEMS and replace it with civetweb in RSB
> instead of just updating the webserver in the RTEMS repo like I
> originally intended.

We should remove the test from RTEMS if there is no code there being tested.

> By the way: Either way civetweb would still need the patches 1 to 4 of
> the original patch set. These four patches only add some rudimentary
> implementations of POSIX functions that RTEMS currently doesn't have.

OK. This comes back to Amar's point we need to manage the process so we 
need to find a solution that works.

>
>>
>>> Theoretically it should be possible to build and install the libcivetweb
>>> without a RTEMS BSP. When building the BSP, we could detect if the
>>> "civetweb.h" file is there using autoconf and either build the mghttpd01
>>> test or not depending on the result. This would allow it to keep an
>>> RTEMS-specific test for civetweb inside the RTEMS repository.
>>
>> I do not wish to see this happen. It is specifically these type of
>> changes I am concerned with because it forces a dependence beyond
>> standards based headers. We need agreed rules here and we need to stick
>> to them.
>>
>> I see we have a structure issue with these types of tests and I think we
>> need find a general solution.
>
> I don't see where this would break rules.

I have not yet agreed to the dependency order being changed. The header 
change allows you to change the order in your private build if you wish 
and that is fine.

> I wouldn't integrate any
> civetweb files or non POSIX headers into newlib.

The ability to build the library without RTEMS changes a dependence 
order and opens up the possibility of a header test check you suggested. 
This is all I am referring too.

> Adding POSIX headers (and only POSIX headers) to newlib would mean that
> we can build civetweb completely independent of the BSP. This allows
> that you _can_  build civetweb before the BSP. But if you prefer you can
> also build it afterward.

Not if the test is in RTEMS as you suggested and you want to build the 
test. Removing the test from RTEMS resolves this point. Where it goes is 
the next problem.

> It would only influence how to run the tests: If civetweb should be
> tested, you would have to build it before the BSP.

This forcing an order on everyone and this is my concern. I do not want 
this to happen and I am fine if you want it and can do it in your own 
builds. My view may change but I would like the newlib header change to 
happen and settle down then we review the situation.

> For the average users it wouldn't make a difference. They most likely
> don't want to test RTEMS but only use it.

Sigh, Joel and I have been working hard over the last few years to open 
testing up so users can test. The RTEMS Project is open and testing is 
an important part and it needs to be open. We want published test 
results and we want users to confirm they can match those results with 
their build. It should not be the domain of experts. My hope is how you 
build for production is how you build to test.

> So it's not relevant for them
> if they build civetweb before or after the BSP.

I do not agree. If a user builds some code they should expect to be able 
to test it. It should be normal and natural. If we are missing tests it 
is something we should fix.

> On the long term, this kind of structure could improve the ability to
> test RTEMS and it's libraries. We can for example build and install
> civetweb and the three network stacks as libraries. If we then build the
> BSP afterward, the build system can detect which libraries are available
> and build a test for each combination (civetweb with each of the network
> stacks). We would just have to link against another library. So for a
> complete test run (including all stacks), you only need one set of libs
> and one BSP built in the right order.

I assume this means the test source becomes a separate package. I had 
also floated the idea and Amar made the point to me that it would be a 
mistake to move the tests and testing away from the code being tested 
and have to agree.

> Of course this would presume that all network stacks use the same
> headers and can be build as libraries. We are not there yet. We still
> have to extract the old network stack and adapt the build system so that
> it would build multiple variations of the test depending on the
> installed libs. I think that at least the extraction of the stack is on
> the wish-list of more than one person on the list. So we are already on
> the way.

With a single vendor source tree you have the ability to make sure all 
parts do build. How the matrix of tests are built against the various 
possible option sets is something I am not sure of.

>
>>
>>> The only problem is: Currently there is no way to build civetweb prior
>>> to the BSP. Civetweb uses some of the network headers that are currently
>>> provided by the BSP.
>>
>> Beyond the standards based headers?
>
> Civetweb can be build with headers from the POSIX standard only.

Great.

> I tried to build civetweb as a multilib and found only one extra POSIX
> header that I missed in the first attempt to move the headers to newlib.
> If we have some more packages that are build this way, it shouldn't take
> too long till we have the most common POSIX headers moved to newlib.

This is great news but may I suggest we put multibs and newlib headers 
to the side and concentrate on the where we put the civetweb and the tests.

Chris