New source layout.

[Amar Takhar]

On 2015-03-10 11:03 +0100, Pavel Pisa wrote:
> Hello Amar and others,
> 
> I would like to express my weak objection to moving
> all includes to single directory. This makes sources
> much less modular is (according to me) backward move.

All input is welcome!

<snip>
> I think that includes really belong near to the implementation and that
> new architecture could be (in optimal case) introduced by adding single
> directory in "arch" subtree.

The includes are near the implementation when they're local headers.  Anything 
that is consumable by another part of RTEMS is moved.


> I think that doing move which is against modularity due to problem
> in build tool is not right decision.

I disagree, see below.  Also to be clear: this move is not being made to solve 
any issues with the build tool.  We are doing it to improve visibility and 
modularity just in a different way from what you are proposing.


> Linux directs INCLUDE_PATH to its common include and per arch include 
> directories + directory for generated files
> 
> Linux make GCC call for build of kernel out of sources tree
> 
> make -f /home/pi/projects/rpi/linux/scripts/Makefile.build obj=fs/aufs
> 
> arm-rpi-linux-gnueabihf-gcc -Wp,-MD,fs/aufs/.cpup.o.d  -nostdinc \
>   -isystem /usr/arm-rpi-linux-gnueabihf/gcc/4.9.1/bin/../lib/gcc/arm-rpi-linux-gnueabihf/4.9.1/include
>   -I/home/pi/projects/rpi/linux/arch/arm/include \
>   -Iarch/arm/include/generated \
>   -I/home/pi/projects/rpi/linux/include -Iinclude \
>   -I/home/pi/projects/rpi/linux/arch/arm/include/uapi \
>   -Iarch/arm/include/generated/uapi \
>   -I/home/pi/projects/rpi/linux/include/uapi \
>   -Iinclude/generated/uapi \
>   -include /home/pi/projects/rpi/linux/include/linux/kconfig.h  \
>   -I/home/pi/projects/rpi/linux/fs/aufs \
>   -Ifs/aufs -D__KERNEL__ -mlittle-endian  \
>   -I/home/pi/projects/rpi/linux/arch/arm/mach-bcm2708/include \
>   ...
>   ...
>   -r -o fs/aufs/aufs.o fs/aufs/module.o fs/aufs/sbinfo.

This is the exact situation we are trying to avoid.  It makes source far header 
to read.  You can also be easily tripped up by having an incorrect order of -I 
switches this is frail.

For example RTEMS has a pthread.h.  In the old build system this was included 
as:

  #include <pthread.h>

When building with waf this gave me many errors because it was picking up the 
compiler <pthread.h>  Why?  Because I 'forgot' an include line 
(-Irtems/posix/).

In waf, this line is now:

  #include <rtems/posix/pthread.h>

This offers resiliency on tripping yourself up with commandlines and it also 
tells you where this file is coming from.  #include <pthread.h> tells you 
absolutely nothing about this.  On my system I've got 3 possible pthread.h to 
include.  My system (FreeBSD), the sparc compiler *and* RTEMS.  Which one is it?


> So for architectures it is my preference to separate includes
> it helps to grep for corresponding function definition
> and declaration and keep them in sync for single arch for example.
> The functions repeats for each arch and if you grep whole tree
> you are lost if you work on single architecture.

OK.


> I even prefer to place exported header files near to implementation
> in the case of libraries because it makes system modular (i.e. if you
> have two implementations for given purpose - small/big, old/new) then
> single enable/disable of one of the libraries flavor directory
> ensures consistent build. On the other hand I agree that headers
> copying has some problems. On sensible systems it can be replaced
> by symlinking and then works great.

This does not solve the problem of fragility.  We tripped up in one BSP.  We had 
two headers:

  HEADER.h
  header.h

These were included in two different BSPs.  The 'preinclude' step copied these 
to the same location -- the contents were different!

What's more, this solution could have never worked on Windows or OS X as it was 
case insensitive.  This bug had been in the repo over 7 years.  Copying and 
symlinking open you up to transparent issues such as this.  RTEMS has over 1,600 
headers.

The proposal I'm making will eliminate 100% of these issues and increase 
visibility when looking at the source file due to absolute includes.


> I have declared some demands on build system when we have been
> in integration phase of OCERA project many years ago and then
> implemented our OMK system based on these rules. Even that we use
> this system for allmost all of our code (Linux apps+driver, RTEMS,
> bare metal, Windows etc), it is dated now and has some significant
> drawbacks, multiple passes etc. But I hope that an requirements
> analysis worth to be read. Keep in mid, that analysis is dated
> (about 2003) on the other hand (i.e CVS -> GIT etc)
<snip> 

We use waf as our make system now all the above requirements are very trivial to 
support.  The waf branch is public which you can see in a previous posting to 
this list.  If there is something not supported that you require please let me 
know.


> http://rtime.felk.cvut.cz/gitweb/rtems-devel.git/tree/HEAD:/rtems-omk-template
> 
>   git clone git://rtime.felk.cvut.cz/rtems-devel.git
>   cd rtems-devel/rtems-omk-template/
>   echo >config.target \
>      RTEMS_MAKEFILE_PATH=/opt/rtems4.11/arm-rtems4.11/lpc17xx_ea_ram 
>   make
> 
> Libraries tracking is not fully functional for RTEMS version in the currenrt
> OMK devel for some reason but rest works well.
> 
> I would be happy if you try to check again what is an optimal solution.
> 
> On the other hand I would be very happy if directory structure
> of RTEMS is updated. Historical (ADA days based) hierarchy
>  c/src/lib/libbsp, c/src/lib/libcpu for subtrees which belongs
> to arch is quite confusing for new users/students from my experience.

Yes it's confusing which is why I wanted to take this opportunity to move it with 
the introduction to the waf build.

Please give it a try the beginning of the thread is here:

  https://lists.rtems.org/pipermail/devel/2015-February/009837.html

The header move I'm proposing has already been done in this repository.  The 
source layout has not changed.

We will be moving towards having 100% Doxygen coverage in all of our source 
files.  This will be the best option for finding areas of the source users are 
looking for.

The move will also help us enforce what is private, public and local.  RTEMS 
will be moving to 100% coverage on all areas of source in the future.  It's 
important to have the distinction of what types of tests we want.  It can be 
tricky to separate what is internally consumable versus a local implementation.  
Anything in include/ is meant to be used more than once.  Moving the files makes 
it very clear within the source itself and while building.

Another benefit of having them all in the same place is it is far easier to see 
what is going on when working on RTEMS.  The preinclude step hides far too much, 
so do multiple -I lines.  Having it all laid out in this way will help us make 
sound choices in the future for how we want interfaces to work.  RTEMS is huge 
it's easy to forget to look at Doxygen the wiki or source files themselves.  
When it's scattered all over the place the 'global picture' is hidden from you.  
The Linux kernel source has different requirements from us.  While their method 
may work for them it is not ideal for RTEMS.

I'm happy to address any concerns with your workflow in the new build system.


Amar.