[PATCH 00/13] Replace mongoose with civetweb.

Fri Apr 29 10:19:57 UTC 2016

Am 29.04.2016 um 09:39 schrieb Chris Johns:
> On 29/04/2016 16:19, Christian Mauderer wrote:
>> Am 29.04.2016 um 02:45 schrieb Chris Johns:
>>> On 28/04/2016 18:27, Christian Mauderer wrote:
>>>> Am 27.04.2016 um 02:07 schrieb Chris Johns:
>>>>> On 26/04/2016 22:22, Christian Mauderer wrote:
>>>>>> Am 26.04.2016 um 04:51 schrieb Chris Johns:
>>>>>>> On 26/04/2016 07:22, Joel Sherrill wrote:
>>>>>>>>
>>>> I put some work into building civetweb using RSB. My configuration is
>>>> now able to build a "libcivetweb.a" using the Makefile of civetweb.
>>>
>>> Excellent. GNU make is available in the RSB on all hosts.
>>>
>>> How do we package this Makefile?
>>
>> The Makefile is the one provided by the civetweb repository. So we
>> shouldn't pack it ourselves.
> 
> Oh sorry, I misunderstood and thought you had written one.
> 
>>
>>>
>>> We have a few things happening we need to land in a coordinated manner.
>>> We have this work, libbsd, LwIP, and the removal of the current stack
>>> from the RTEMS source tree. If we do not attempt to coordinate these
>>> pieces work will be repeated, code duplicated and users will be given a
>>> confusing inconsistent view of networking.
>>>
>>> What you have done here is really important and you are breaking new
>>> ground. We also have a GSoC project under-way for LwIP and Pavel and I
>>> have been discussing that work. There are a number of strikingly similar
>>> issues appearing.
>>>
>>> We are currently thinking of a repo for LwIP on git.rtems.org, eg
>>> rtems-lwip.git. It will contain some build system support files,
>>> license, a readme (I suppose) and a submodule reference to the upstream
>>> LwIP repo. The submodule would be a specific hash of the upstream
>>> project. At release time the release process creates a single tarball of
>>> all source, I do this for libbsd now.
>>>
>>> Do you think this would work for civetweb?
>>
>> I'm not sure if this would be the right way. This sounds more like
>> forking civetweb. This seems overkill for only a few patches that
>> hopefully can go away in the future.
> 
> I misunderstood and thought you had build system files.
> 
>> If we want to keep a own repository
>> up to date with the upstream repository this means an ongoing effort.
> 
> I had a long chat to Amar about this topic today and he told me we
> should follow the FreeBSD model for vendor software. FreeBSD has
> https://svnweb.freebsd.org/base/vendor/ and it contains the vendor
> source untouched and then the version used is a copy with local changes.
> If you follow the link the bind directory is the untouched vendor source
> and bind9 is a copy of the bind v9 vendor source with the local changes.
> His view is we create rtems-vendor.git and bring all the code into this
> repo. He said we should bring all the networking stacks and libbsd each
> as a separate directory. He said it brings into a single place the
> history of all the packages we have to manage and it lets us combine
> this code with the tests. It means users have a single place to go for
> any vendor source. It is a change in direction from the RSB per package
> approach but if we adopt rtems-vendor.git I do not mind. It also allows
> the testing framework he has been working to be used. I will let him
> explain this more as it is not area I know too much about.
> 
>>
>>>
>>> What is not clear to me is do we following the LwIP path and a separate
>>> repo for this package or do we have an 'rtems-net-services.git' repo
>>> this code is part of? A combined repo would mean the build system used
>>> is either a mix based on each part or we unify it to a common build
>>> system. In time we will have other pieces of code added, eg telnetd etc
>>> as we moved that from the in source tree.
>>>
>>> I see Sebastian removed the unreferenced services directory from libbsd
>>> this week (the vc email was too big for the list) and thank you
>>> Sebastian. It did make me wonder how we manage these things.
>>>
>>> We also need to unify the network configuration for testing. Currently
>>> libbsd is inconsistent. Some tests use the waf configure command line
>>> settings and others do not and force DHCP. I am looking to change this
>>> so we can have a better solution that allows more complex
>>> configurations. The intention is this can be used for LwIP as well. I
>>> will post my ideas when I get time.
>>>
>>>> The next steps would be to remove mghttpd from RTEMS. Currently that
>>>> means that I have to remove the test too. Any ideas where we could move
>>>> it to? If we don't have a solution: Do we really want to leave the code
>>>> untested until someone has the time and budget to implement a external
>>>> test suit?
>>>
>>> This is a really good question. I do not know and it also relates to how
>>> we package this code.
>>
>> This is a question at hand. So I don't think that we can avoid it if we
>> want to remove mongoose from RTEMS and replace it with civetweb in RSB
>> instead of just updating the webserver in the RTEMS repo like I
>> originally intended.
> 
> We should remove the test from RTEMS if there is no code there being
> tested.

So we would replace a tested code with some untested one. Not an ideal
solution.

> 
>> By the way: Either way civetweb would still need the patches 1 to 4 of
>> the original patch set. These four patches only add some rudimentary
>> implementations of POSIX functions that RTEMS currently doesn't have.
> 
> OK. This comes back to Amar's point we need to manage the process so we
> need to find a solution that works.
> 

What I wanted to say was, that we should handle these four patches
independent of the other discussion. They are just normal patches that
improve RTEMS. They allow some additional applications to build and
link. One application is civetweb. But I'm sure there are others too and
the patches can be useful regardless how we handle the ones that concern
mongoose / civetweb.

>>
>>>
>>>> Theoretically it should be possible to build and install the
>>>> libcivetweb
>>>> without a RTEMS BSP. When building the BSP, we could detect if the
>>>> "civetweb.h" file is there using autoconf and either build the
>>>> mghttpd01
>>>> test or not depending on the result. This would allow it to keep an
>>>> RTEMS-specific test for civetweb inside the RTEMS repository.
>>>
>>> I do not wish to see this happen. It is specifically these type of
>>> changes I am concerned with because it forces a dependence beyond
>>> standards based headers. We need agreed rules here and we need to stick
>>> to them.
>>>
>>> I see we have a structure issue with these types of tests and I think we
>>> need find a general solution.
>>
>> I don't see where this would break rules.
> 
> I have not yet agreed to the dependency order being changed. The header
> change allows you to change the order in your private build if you wish
> and that is fine.
> 
>> I wouldn't integrate any
>> civetweb files or non POSIX headers into newlib.
> 
> The ability to build the library without RTEMS changes a dependence
> order and opens up the possibility of a header test check you suggested.
> This is all I am referring too.
> 
>> Adding POSIX headers (and only POSIX headers) to newlib would mean that
>> we can build civetweb completely independent of the BSP. This allows
>> that you _can_  build civetweb before the BSP. But if you prefer you can
>> also build it afterward.
> 
> Not if the test is in RTEMS as you suggested and you want to build the
> test. Removing the test from RTEMS resolves this point. Where it goes is
> the next problem.

A problem that obviously isn't solved yet and can't be easily solved on
the short term. So from my point of view this blocks moving mongoose /
civetweb (or any other third party package like the old network stack)
out of the RTEMS tree until we found a solution that is OK for all of
the community.

> 
>> It would only influence how to run the tests: If civetweb should be
>> tested, you would have to build it before the BSP.
> 
> This forcing an order on everyone and this is my concern. I do not want
> this to happen and I am fine if you want it and can do it in your own
> builds. My view may change but I would like the newlib header change to
> happen and settle down then we review the situation.
> 
>> For the average users it wouldn't make a difference. They most likely
>> don't want to test RTEMS but only use it.
> 
> Sigh, Joel and I have been working hard over the last few years to open
> testing up so users can test. The RTEMS Project is open and testing is
> an important part and it needs to be open. We want published test
> results and we want users to confirm they can match those results with
> their build. It should not be the domain of experts. My hope is how you
> build for production is how you build to test.

It wasn't my intention to damage any effort. It would be great to
simplify the test process so that everyone can do it. Thanks for that.

> 
>> So it's not relevant for them
>> if they build civetweb before or after the BSP.
> 
> I do not agree. If a user builds some code they should expect to be able
> to test it. It should be normal and natural. If we are missing tests it
> is something we should fix.
> 
>> On the long term, this kind of structure could improve the ability to
>> test RTEMS and it's libraries. We can for example build and install
>> civetweb and the three network stacks as libraries. If we then build the
>> BSP afterward, the build system can detect which libraries are available
>> and build a test for each combination (civetweb with each of the network
>> stacks). We would just have to link against another library. So for a
>> complete test run (including all stacks), you only need one set of libs
>> and one BSP built in the right order.
> 
> I assume this means the test source becomes a separate package. I had
> also floated the idea and Amar made the point to me that it would be a
> mistake to move the tests and testing away from the code being tested
> and have to agree.

I agree that it's not a good idea if the test code and the tested code
is separated.

> 
>> Of course this would presume that all network stacks use the same
>> headers and can be build as libraries. We are not there yet. We still
>> have to extract the old network stack and adapt the build system so that
>> it would build multiple variations of the test depending on the
>> installed libs. I think that at least the extraction of the stack is on
>> the wish-list of more than one person on the list. So we are already on
>> the way.
> 
> With a single vendor source tree you have the ability to make sure all
> parts do build. How the matrix of tests are built against the various
> possible option sets is something I am not sure of.
> 
>>
>>>
>>>> The only problem is: Currently there is no way to build civetweb prior
>>>> to the BSP. Civetweb uses some of the network headers that are
>>>> currently
>>>> provided by the BSP.
>>>
>>> Beyond the standards based headers?
>>
>> Civetweb can be build with headers from the POSIX standard only.
> 
> Great.
> 
>> I tried to build civetweb as a multilib and found only one extra POSIX
>> header that I missed in the first attempt to move the headers to newlib.
>> If we have some more packages that are build this way, it shouldn't take
>> too long till we have the most common POSIX headers moved to newlib.
> 
> This is great news but may I suggest we put multibs and newlib headers
> to the side and concentrate on the where we put the civetweb and the tests.
> 

I'm OK with taking this part out of the discussion. The headers
influence the possible build order (without them building civetweb
before the BSP isn't possible) but they don't have to do anything with
the testing problem.

> Chris

-- 
--------------------------------------------
embedded brains GmbH
Christian Mauderer
Dornierstr. 4
D-82178 Puchheim
Germany
email: christian.mauderer at embedded-brains.de
Phone: +49-89-18 94 741 - 18
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.