[PATCH 00/13] Replace mongoose with civetweb.

Fri Apr 29 06:19:05 UTC 2016

Am 29.04.2016 um 02:45 schrieb Chris Johns:
> On 28/04/2016 18:27, Christian Mauderer wrote:
>> Am 27.04.2016 um 02:07 schrieb Chris Johns:
>>> On 26/04/2016 22:22, Christian Mauderer wrote:
>>>> Am 26.04.2016 um 04:51 schrieb Chris Johns:
>>>>> On 26/04/2016 07:22, Joel Sherrill wrote:
>>>>>>
>> I put some work into building civetweb using RSB. My configuration is
>> now able to build a "libcivetweb.a" using the Makefile of civetweb.
> 
> Excellent. GNU make is available in the RSB on all hosts.
> 
> How do we package this Makefile?

The Makefile is the one provided by the civetweb repository. So we
shouldn't pack it ourselves.

> 
> We have a few things happening we need to land in a coordinated manner.
> We have this work, libbsd, LwIP, and the removal of the current stack
> from the RTEMS source tree. If we do not attempt to coordinate these
> pieces work will be repeated, code duplicated and users will be given a
> confusing inconsistent view of networking.
> 
> What you have done here is really important and you are breaking new
> ground. We also have a GSoC project under-way for LwIP and Pavel and I
> have been discussing that work. There are a number of strikingly similar
> issues appearing.
> 
> We are currently thinking of a repo for LwIP on git.rtems.org, eg
> rtems-lwip.git. It will contain some build system support files,
> license, a readme (I suppose) and a submodule reference to the upstream
> LwIP repo. The submodule would be a specific hash of the upstream
> project. At release time the release process creates a single tarball of
> all source, I do this for libbsd now.
> 
> Do you think this would work for civetweb?

I'm not sure if this would be the right way. This sounds more like
forking civetweb. This seems overkill for only a few patches that
hopefully can go away in the future. If we want to keep a own repository
up to date with the upstream repository this means an ongoing effort.

> 
> What is not clear to me is do we following the LwIP path and a separate
> repo for this package or do we have an 'rtems-net-services.git' repo
> this code is part of? A combined repo would mean the build system used
> is either a mix based on each part or we unify it to a common build
> system. In time we will have other pieces of code added, eg telnetd etc
> as we moved that from the in source tree.
> 
> I see Sebastian removed the unreferenced services directory from libbsd
> this week (the vc email was too big for the list) and thank you
> Sebastian. It did make me wonder how we manage these things.
> 
> We also need to unify the network configuration for testing. Currently
> libbsd is inconsistent. Some tests use the waf configure command line
> settings and others do not and force DHCP. I am looking to change this
> so we can have a better solution that allows more complex
> configurations. The intention is this can be used for LwIP as well. I
> will post my ideas when I get time.
> 
>> The next steps would be to remove mghttpd from RTEMS. Currently that
>> means that I have to remove the test too. Any ideas where we could move
>> it to? If we don't have a solution: Do we really want to leave the code
>> untested until someone has the time and budget to implement a external
>> test suit?
> 
> This is a really good question. I do not know and it also relates to how
> we package this code.

This is a question at hand. So I don't think that we can avoid it if we
want to remove mongoose from RTEMS and replace it with civetweb in RSB
instead of just updating the webserver in the RTEMS repo like I
originally intended.

By the way: Either way civetweb would still need the patches 1 to 4 of
the original patch set. These four patches only add some rudimentary
implementations of POSIX functions that RTEMS currently doesn't have.

> 
>> Theoretically it should be possible to build and install the libcivetweb
>> without a RTEMS BSP. When building the BSP, we could detect if the
>> "civetweb.h" file is there using autoconf and either build the mghttpd01
>> test or not depending on the result. This would allow it to keep an
>> RTEMS-specific test for civetweb inside the RTEMS repository.
> 
> I do not wish to see this happen. It is specifically these type of
> changes I am concerned with because it forces a dependence beyond
> standards based headers. We need agreed rules here and we need to stick
> to them.
> 
> I see we have a structure issue with these types of tests and I think we
> need find a general solution.

I don't see where this would break rules. I wouldn't integrate any
civetweb files or non POSIX headers into newlib.

Adding POSIX headers (and only POSIX headers) to newlib would mean that
we can build civetweb completely independent of the BSP. This allows
that you _can_  build civetweb before the BSP. But if you prefer you can
also build it afterward.

It would only influence how to run the tests: If civetweb should be
tested, you would have to build it before the BSP.

For the average users it wouldn't make a difference. They most likely
don't want to test RTEMS but only use it. So it's not relevant for them
if they build civetweb before or after the BSP.

On the long term, this kind of structure could improve the ability to
test RTEMS and it's libraries. We can for example build and install
civetweb and the three network stacks as libraries. If we then build the
BSP afterward, the build system can detect which libraries are available
and build a test for each combination (civetweb with each of the network
stacks). We would just have to link against another library. So for a
complete test run (including all stacks), you only need one set of libs
and one BSP built in the right order.

Of course this would presume that all network stacks use the same
headers and can be build as libraries. We are not there yet. We still
have to extract the old network stack and adapt the build system so that
it would build multiple variations of the test depending on the
installed libs. I think that at least the extraction of the stack is on
the wish-list of more than one person on the list. So we are already on
the way.

> 
>> The only problem is: Currently there is no way to build civetweb prior
>> to the BSP. Civetweb uses some of the network headers that are currently
>> provided by the BSP.
> 
> Beyond the standards based headers?

Civetweb can be build with headers from the POSIX standard only.

I tried to build civetweb as a multilib and found only one extra POSIX
header that I missed in the first attempt to move the headers to newlib.
If we have some more packages that are build this way, it shouldn't take
too long till we have the most common POSIX headers moved to newlib.

Kind regards

Christian

> 
>> So the process would extend to the following:
>> - build BSP (without mghttpd01 test)
>> - build civetweb
>> - re-build BSP (this time with mghttpd01 test)
>> - run tests
>>
>> This could be solved by moving the network headers into newlib like
>> suggested in the other thread of discussion.
> 
> Lets figure out a suitable scalable architecture for network services
> like this then come back and look at the detail. This is just one package.
> 
> Chris

-- 
--------------------------------------------
embedded brains GmbH
Christian Mauderer
Dornierstr. 4
D-82178 Puchheim
Germany
email: christian.mauderer at embedded-brains.de
Phone: +49-89-18 94 741 - 18
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.