Untar: Memory problem with big files (800k) in RTEMS filesystem: simple trial!]
Joel Sherrill
joel.sherrill at OARcorp.com
Fri Feb 11 15:07:33 UTC 2000
Jake Janovetz wrote:
>
> Oscar,
>
> As I recall, Joel set the maximum file size (through a
> heirarchy of three blocks) to something like 278kBytes. Have
> you overridden this yet? Untar was written without a whole
> lot of error-checking, so it probably won't tell you specifically
> what went wrong.
I believe the IMFS should return the correct error though.
> If you look in c/src/lib/libc/imfs.h, you can find how the
> file size relies on the number of block slots and the number of
> bytes per block. If you've already done all this, I can take
> another look at the untar code and test it a bit with bigger files.
What is the right value for the size of each block. WIth the IMFS,
the max file size is limited by the size of each block. Each file
is represented by a single, doubly and triply indirect set of block
pointers. The number of pointers in a block is based on this block
size factor.
How large is reasonable? On one hand, you don't want to waste a lot
of memory on small files. On the other, setting it too low causes
problems with file sizes. Here is a list of various block sizes
and the maximum file size:
max_filesize with blocks of 16 is 1328
max_filesize with blocks of 32 is 18656
max_filesize with blocks of 64 is 279488
max_filesize with blocks of 128 is 4329344
max_filesize with blocks of 256 is 68173568
Basically it is a tradeoff between too small, wasting memory, and
being able to test the max file size error. :)
I recall thinking 128 was a pretty reasonable number when I
wasn't trying to generate the error condition.
Comments?
> Cheers,
> Jake
>
> On Fri, Feb 11, 2000 at 11:16:08AM +0100, Oscar Martinez de la Torre wrote:
> > Hi!
> >
> > I'm trying to use Kaffe with RTEMS on ERC32.
> >
> > As I need a filesystem with the .class file and Klasses.jar
> > I tar them and ld and link with the executable (Kaffe0.exe)
> > But I have found several problems because of the
> > untar function and memory.
> >
> > Is anyone working with filesystem and big files (1Mb)?
> > I'm using linux and RPMs of snapshot 19991203!
> >
> > Below, I tell you more details!
> >
> > Thanks,
> > Oscar
> >
> >
> >
> > In order to test and isolate the problem, I have taken
> > a simple program (test.c I attach it) which works pretty good
> > It untars, reads and write a small file (hola)
> > So, I haved added several files which I use with Kaffe!
> >
> > -rwxr-xr-x 1 omt wms 2601580 Feb 11 10:06 Kaffe0.exe
> > -rw-r--r-- 1 omt wms 885137 Feb 11 10:05 Klasses.jar
> > -rwxr-xr-x 1 omt wms 5 Jan 27 14:54 hola*
> >
> > The settings are:
> >
> > <confdefs.h>
> > #define CONFIGURE_EXTRA_TASK_STACKS 1000000
> >
> > ( I tried
> > #define CONFIGURE_EXTRA_TASK_STACKS 4000000
> > but
> > bspstart: Not enough RAM!!!)
> >
> > I have overridden the default values:
> >
> > /opt/rtems/sparc-rtems/lib/linkcmds
> > /* Default values, can be overridden */
> > _PROM_SIZE = 8M;
> > _RAM_SIZE = 32M;
> >
> > I have made 2 trials: First, if I do like this:
> >
> > venus:~/TEST3$ tar cvf tarfile Kaffe0.exe Klasses.jar hola
> > venus:~/TEST3$ sparc-rtems-ld -r -o tarfile.o -b binary tarfile
> > venus:~/TEST3$ /opt/rtems/bin/sparc-rtems-gcc --pipe
> > -B/opt/rtems//erc32/lib/ -specs bsp_specs -qrtems -g -Wall -ansi -fasm
> > -O4 -mcpu=cypress -c -I/opt/rtems/sparc-rtems/include/ -o
> > test.o test.c
> > venus:~/TEST3$ /opt/rtems/bin/sparc-rtems-gcc --pipe
> > -B/opt/rtems//erc32/lib/ -specs bsp_specs -qrtems -g -Wall -ansi -fasm
> > -O4 -mcpu=cypress test.o tarfile.o -o test.exe
> >
> >
> > venus:~/TEST3$ /usr/local/erc32/bin/sis -ram 32000 -rom 8000 test.exe
> > SIS - SPARC instruction simulator 3.0.5, copyright Jiri Gaisler
> > 1995-1998
> > Bug-reports to jgais at ws.estec.esa.nl
> > serial port A on stdin/stdout
> > allocated 32768 K RAM memory
> > allocated 8192 K ROM memory
> > loading:
> > section .text at 0x02000000 (106416 bytes)
> > section .data at 0x02019fb0 (3494112 bytes)
> > section .bss at 0x0236f090 (22124 bytes)(not loaded)
> > sis> run
> > IU in error mode (257)
> > 16 00000000 00000000 unimp 0
> >
> >
> > venus:~/TEST3$ /usr/local/erc32/bin/sparc-rtems-gdb
> > test.exe
> > GNU gdb 4.17.gnat.3.11p
> > Copyright 1998 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> > are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB. Type "show warranty" for
> > details.
> > This GDB was configured as "--host=i686-pc-linux-gnu
> > --target=sparc-erc32-elf32"...
> > (gdb) tar sim -ram 32000 -rom 8000
> >
> > SIS - SPARC instruction simulator 3.0.5
> > Bug-reports to Jiri Gaisler ESA/ESTEC (jgais at ws.estec.esa.nl)
> > serial port A on stdin/stdout
> > allocated 32768 K RAM memory
> > allocated 8192 K ROM memory
> > Connected to the simulator.
> > (gdb) lo
> > loading:
> > section .text at 0x02000000 (106416 bytes)
> > section .data at 0x02019fb0 (3494112 bytes)
> > section .bss at 0x0236f090 (22124 bytes)(not loaded)
> > (gdb) bre Init
> > Breakpoint 1 at 0x2001238: file test.c, line 70.
> > (gdb)
> > Note: breakpoint 1 also set at pc 0x2001238.
> > Breakpoint 2 at 0x2001238: file test.c, line 70.
> > (gdb) n
> > Cannot find bounds of current function
> > (gdb) run
> > Starting program: /home/omt/TEST3/test.exe
> > loading:
> > section .text at 0x02000000 (106416 bytes)
> > section .data at 0x02019fb0 (3494112 bytes)
> > section .bss at 0x0236f090 (22124 bytes)(not loaded)
> > Breakpoint 1, Init (argument=0) at test.c:70
> > 70 char prueba[5]="rtems";
> > 77 status = Untar_FromMemory((unsigned char
> > *)(&_binary_tarfile_start), (unsigned long) &_binary_tarfile_size);
> > (gdb) n
> > assertion "memory" failed: file
> > "../../../../../../rtems-19991203.joel/c/src/lib/libc/memfile.c", line
> > 340
> > Program exited normally.
> >
> > rtems-19991203.joel/c/src/lib/libc/memfile.c line 340:
> > memory = memfile_alloc_block();
> > assert( memory );
> > if ( !memory )
> > return 1;
> > *block_entry_ptr = memory;
> > return 0;
> > }
> >
> >
> >
> >
> > If I only use Klasses.jar
> >
> > venus:~/TEST3$ tar cvf tarfile Klasses.jar hola
> > -rw-r--r-- 1 omt wms 890880 Feb 11 10:18 tarfile
> > -rw-r--r-- 1 omt wms 891360 Feb 11 10:19 tarfile.o
> >
> > venus:~/TEST3$ /usr/local/erc32/bin/sparc-rtems-gdb
> > test.exe
> > GNU gdb 4.17.gnat.3.11p
> > Copyright 1998 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> > are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB. Type "show warranty" for
> > details.
> > This GDB was configured as "--host=i686-pc-linux-gnu
> > --target=sparc-erc32-elf32"...
> > (gdb) tar sim -ram 32000 -rom 8000
> > SIS - SPARC instruction simulator 3.0.5
> > Bug-reports to Jiri Gaisler ESA/ESTEC (jgais at ws.estec.esa.nl)
> > serial port A on stdin/stdout
> > allocated 32768 K RAM memory
> > allocated 8192 K ROM memory
> > Connected to the simulator.
> > (gdb) lo
> > loading:
> > section .text at 0x02000000 (106416 bytes)
> > section .data at 0x02019fb0 (893152 bytes)
> > section .bss at 0x020f4090 (22124 bytes)(not loaded)
> > (gdb) bre Init
> > Breakpoint 1 at 0x2001238: file test.c, line 70.
> > 77 status = Untar_FromMemory((unsigned char
> > *)(&_binary_tarfile_start), (unsigned long) &_binary_tarfile_size);
> > (gdb)
> > Error during write
> >
> > Untar_FromMemory (tar_buf=0x201a0e0 "Klasses.jar", size=890880) at
> > ../../../../../../rtems-19991203.joel/c/src/libmisc/untar/untar.c:93
> >
> > So, the same problem I send you yesterday night by going inside untar:
> >
> > 215 for (i=0; i<nblocks; i++)
> > (gdb)
> > 217 len = ((sizeToGo < 512L)?(sizeToGo):(512L));
> > (gdb)
> > 218 n = fwrite(&tar_buf[ptr], 1, len, fp);
> > (gdb)
> > 219 if (n != len)
> > (gdb)
> > 224 ptr += 512;
> >
> > ++++++++++++++
> > This is a loop
> > ++++++++++++++
> >
> > (gdb) print nblocks
> > $118 = 1729
> > (gdb) print i
> > $119 = 11
> >
> > it needs 5 next for i=i+1
> > so, I try i=i+1000
> >
> > (gdb) next 5000
> > Error during write
> >
> >
> > I don't know how to solve the problem...
> >
> > Thanks for your support,
> > Oscar
> > --
> > Óscar Martínez de la Torre - Office Phone: +31-(0)71.565.5844
> > Telecommunication Engineer at Universidad Politécnica de Madrid
> > European Space Agency ESTEC/TOS/EMS Erasmus Building Office Nb212
> > Keplerlaan,1 - NL 2201 AZ - Noordwijk ZH - The Netherlands
> > Mail: omt at wm.estec.esa.nl - Personal Mail: omt at etsit.upm.es
> > /* Init
> > * This routine is the initialization task for this test program.
> > */
> >
> > #include <bsp.h>
> >
> > #define CONFIGURE_TEST_NEEDS_CONSOLE_DRIVER
> > #define CONFIGURE_TEST_NEEDS_CLOCK_DRIVER
> > #define CONFIGURE_RTEMS_INIT_TASKS_TABLE
> >
> > #define CONFIGURE_EXECUTIVE_RAM_SIZE (512*1024)
> > #define CONFIGURE_MAXIMUM_SEMAPHORES 20
> > #define CONFIGURE_MAXIMUM_TASKS 20
> >
> > #define CONFIGURE_INIT_TASK_STACK_SIZE (10*1024)
> > #define CONFIGURE_INIT_TASK_PRIORITY 120
> > #define CONFIGURE_INIT_TASK_INITIAL_MODES (RTEMS_PREEMPT | \
> > RTEMS_NO_TIMESLICE | \
> > RTEMS_NO_ASR | \
> > RTEMS_INTERRUPT_LEVEL(0))
> >
> > #define CONFIGURE_INIT
> >
> > rtems_task Init(
> > rtems_task_argument argument
> > );
> >
> > /* configuration information */
> >
> > #include <confdefs.h>
> >
> > #include <errno.h>
> > #include <time.h>
> >
> > #include <confdefs.h>
> > #include <stdio.h>
> > #include <rtems/rtems_bsdnet.h>
> > #include <ftpd.h>
> >
> > #include <unistd.h>
> >
> > #include <rtems/error.h>
> > #include <rpc/rpc.h>
> > #include <netinet/in.h>
> > #include <time.h>
> >
> > #include <arpa/inet.h>
> > #include <sys/socket.h>
> >
> > #include <fcntl.h>
> > #include <rtems/untar.h>
> >
> > #define PERMS 0666
> >
> >
> > extern int _binary_tarfile_start;
> > extern int _binary_tarfile_size;
> >
> > rtems_task Init(
> > rtems_task_argument argument
> > )
> >
> > {
> >
> > rtems_status_code status;
> > int fd;
> > int fd2;
> > int ok;
> > char buffer[5];
> > char prueba[5]="rtems";
> >
> >
> > printf( "START\n" );
> >
> > printf("%s\n",prueba);
> >
> > status = Untar_FromMemory((unsigned char *)(&_binary_tarfile_start), (unsigned long) &_binary_tarfile_size);
> >
> > fd = open("hola",O_RDWR);
> > if (fd==-1)
> > printf("problems open: %i\n",fd);
> >
> > while ((ok = read(fd, buffer, 4)) >0)
> > printf("n:%i leo:%s\n",ok,buffer);
> >
> > ok = close(fd);
> > fd = open("hola",O_RDWR);
> > if (fd==-1)
> > printf("problems open: %i\n",fd);
> >
> > ok=write(fd,prueba,4);
> > if (ok==-1)
> > printf("problems w1: %i\n",ok);
> >
> > ok = close(fd);
> > fd = open("hola",O_RDWR);
> >
> > while ((ok = read(fd, buffer, 4)) >0)
> > printf("2n:%i leo:%s\n",ok,buffer);
> >
> > fd2 = creat("adios",PERMS);
> > if (fd2==-1)
> > printf("problems creat: %i\n",fd);
> >
> > ok=write(fd2,prueba,4);
> > if (ok==-1)
> > printf("problems w2: %i\n",ok);
> >
> > ok=read(fd2,buffer,4);
> > if (ok==-1)
> > printf("problems r2: %i\n",ok);
> >
> > printf("%s\n",buffer);
> > printf("%s\n",prueba);
> >
> > exit(0);
> > }
> >
>
> --
> janovetz at uiuc.edu | How can it be that mathematics, being after all a
> University of Illinois | product of human thought independent of experience,
> | is so admirably adapted to the objects of reality?
> PP-ASEL | - Albert Einstein
>
> Disclaimer: The policies of this University certainly do not reflect my
> own opinions, objectives, or agenda.
--
Joel Sherrill, Ph.D. Director of Research & Development
joel at OARcorp.com On-Line Applications Research
Ask me about RTEMS: a free RTOS Huntsville AL 35805
Support Available (256) 722-9985
More information about the users
mailing list