Untar: Memory problem with big files (800k) in RTEMS filesystem: simple trial!]
Jake Janovetz
janovetz at tempest.ece.uiuc.edu
Fri Feb 11 16:06:11 UTC 2000
Joel,
I think 128 is certainly reasonable. On the other hand,
I don't think 256 is completely unreasonable. 64 is a bit small.
Cheers,
Jake
On Fri, Feb 11, 2000 at 09:07:33AM -0600, Joel Sherrill wrote:
> Jake Janovetz wrote:
> >
> > Oscar,
> >
> > As I recall, Joel set the maximum file size (through a
> > heirarchy of three blocks) to something like 278kBytes. Have
> > you overridden this yet? Untar was written without a whole
> > lot of error-checking, so it probably won't tell you specifically
> > what went wrong.
>
> I believe the IMFS should return the correct error though.
>
> > If you look in c/src/lib/libc/imfs.h, you can find how the
> > file size relies on the number of block slots and the number of
> > bytes per block. If you've already done all this, I can take
> > another look at the untar code and test it a bit with bigger files.
>
> What is the right value for the size of each block. WIth the IMFS,
> the max file size is limited by the size of each block. Each file
> is represented by a single, doubly and triply indirect set of block
> pointers. The number of pointers in a block is based on this block
> size factor.
>
> How large is reasonable? On one hand, you don't want to waste a lot
> of memory on small files. On the other, setting it too low causes
> problems with file sizes. Here is a list of various block sizes
> and the maximum file size:
>
> max_filesize with blocks of 16 is 1328
> max_filesize with blocks of 32 is 18656
> max_filesize with blocks of 64 is 279488
> max_filesize with blocks of 128 is 4329344
> max_filesize with blocks of 256 is 68173568
>
> Basically it is a tradeoff between too small, wasting memory, and
> being able to test the max file size error. :)
>
> I recall thinking 128 was a pretty reasonable number when I
> wasn't trying to generate the error condition.
>
> Comments?
>
>
>
> > Cheers,
> > Jake
> >
> > On Fri, Feb 11, 2000 at 11:16:08AM +0100, Oscar Martinez de la Torre wrote:
> > > Hi!
> > >
> > > I'm trying to use Kaffe with RTEMS on ERC32.
> > >
> > > As I need a filesystem with the .class file and Klasses.jar
> > > I tar them and ld and link with the executable (Kaffe0.exe)
> > > But I have found several problems because of the
> > > untar function and memory.
> > >
> > > Is anyone working with filesystem and big files (1Mb)?
> > > I'm using linux and RPMs of snapshot 19991203!
> > >
> > > Below, I tell you more details!
> > >
> > > Thanks,
> > > Oscar
> > >
> > >
> > >
> > > In order to test and isolate the problem, I have taken
> > > a simple program (test.c I attach it) which works pretty good
> > > It untars, reads and write a small file (hola)
> > > So, I haved added several files which I use with Kaffe!
> > >
> > > -rwxr-xr-x 1 omt wms 2601580 Feb 11 10:06 Kaffe0.exe
> > > -rw-r--r-- 1 omt wms 885137 Feb 11 10:05 Klasses.jar
> > > -rwxr-xr-x 1 omt wms 5 Jan 27 14:54 hola*
> > >
> > > The settings are:
> > >
> > > <confdefs.h>
> > > #define CONFIGURE_EXTRA_TASK_STACKS 1000000
> > >
> > > ( I tried
> > > #define CONFIGURE_EXTRA_TASK_STACKS 4000000
> > > but
> > > bspstart: Not enough RAM!!!)
> > >
> > > I have overridden the default values:
> > >
> > > /opt/rtems/sparc-rtems/lib/linkcmds
> > > /* Default values, can be overridden */
> > > _PROM_SIZE = 8M;
> > > _RAM_SIZE = 32M;
> > >
> > > I have made 2 trials: First, if I do like this:
> > >
> > > venus:~/TEST3$ tar cvf tarfile Kaffe0.exe Klasses.jar hola
> > > venus:~/TEST3$ sparc-rtems-ld -r -o tarfile.o -b binary tarfile
> > > venus:~/TEST3$ /opt/rtems/bin/sparc-rtems-gcc --pipe
> > > -B/opt/rtems//erc32/lib/ -specs bsp_specs -qrtems -g -Wall -ansi -fasm
> > > -O4 -mcpu=cypress -c -I/opt/rtems/sparc-rtems/include/ -o
> > > test.o test.c
> > > venus:~/TEST3$ /opt/rtems/bin/sparc-rtems-gcc --pipe
> > > -B/opt/rtems//erc32/lib/ -specs bsp_specs -qrtems -g -Wall -ansi -fasm
> > > -O4 -mcpu=cypress test.o tarfile.o -o test.exe
> > >
> > >
> > > venus:~/TEST3$ /usr/local/erc32/bin/sis -ram 32000 -rom 8000 test.exe
> > > SIS - SPARC instruction simulator 3.0.5, copyright Jiri Gaisler
> > > 1995-1998
> > > Bug-reports to jgais at ws.estec.esa.nl
> > > serial port A on stdin/stdout
> > > allocated 32768 K RAM memory
> > > allocated 8192 K ROM memory
> > > loading:
> > > section .text at 0x02000000 (106416 bytes)
> > > section .data at 0x02019fb0 (3494112 bytes)
> > > section .bss at 0x0236f090 (22124 bytes)(not loaded)
> > > sis> run
> > > IU in error mode (257)
> > > 16 00000000 00000000 unimp 0
> > >
> > >
> > > venus:~/TEST3$ /usr/local/erc32/bin/sparc-rtems-gdb
> > > test.exe
> > > GNU gdb 4.17.gnat.3.11p
> > > Copyright 1998 Free Software Foundation, Inc.
> > > GDB is free software, covered by the GNU General Public License, and you
> > > are
> > > welcome to change it and/or distribute copies of it under certain
> > > conditions.
> > > Type "show copying" to see the conditions.
> > > There is absolutely no warranty for GDB. Type "show warranty" for
> > > details.
> > > This GDB was configured as "--host=i686-pc-linux-gnu
> > > --target=sparc-erc32-elf32"...
> > > (gdb) tar sim -ram 32000 -rom 8000
> > >
> > > SIS - SPARC instruction simulator 3.0.5
> > > Bug-reports to Jiri Gaisler ESA/ESTEC (jgais at ws.estec.esa.nl)
> > > serial port A on stdin/stdout
> > > allocated 32768 K RAM memory
> > > allocated 8192 K ROM memory
> > > Connected to the simulator.
> > > (gdb) lo
> > > loading:
> > > section .text at 0x02000000 (106416 bytes)
> > > section .data at 0x02019fb0 (3494112 bytes)
> > > section .bss at 0x0236f090 (22124 bytes)(not loaded)
> > > (gdb) bre Init
> > > Breakpoint 1 at 0x2001238: file test.c, line 70.
> > > (gdb)
> > > Note: breakpoint 1 also set at pc 0x2001238.
> > > Breakpoint 2 at 0x2001238: file test.c, line 70.
> > > (gdb) n
> > > Cannot find bounds of current function
> > > (gdb) run
> > > Starting program: /home/omt/TEST3/test.exe
> > > loading:
> > > section .text at 0x02000000 (106416 bytes)
> > > section .data at 0x02019fb0 (3494112 bytes)
> > > section .bss at 0x0236f090 (22124 bytes)(not loaded)
> > > Breakpoint 1, Init (argument=0) at test.c:70
> > > 70 char prueba[5]="rtems";
> > > 77 status = Untar_FromMemory((unsigned char
> > > *)(&_binary_tarfile_start), (unsigned long) &_binary_tarfile_size);
> > > (gdb) n
> > > assertion "memory" failed: file
> > > "../../../../../../rtems-19991203.joel/c/src/lib/libc/memfile.c", line
> > > 340
> > > Program exited normally.
> > >
> > > rtems-19991203.joel/c/src/lib/libc/memfile.c line 340:
> > > memory = memfile_alloc_block();
> > > assert( memory );
> > > if ( !memory )
> > > return 1;
> > > *block_entry_ptr = memory;
> > > return 0;
> > > }
> > >
> > >
> > >
> > >
> > > If I only use Klasses.jar
> > >
> > > venus:~/TEST3$ tar cvf tarfile Klasses.jar hola
> > > -rw-r--r-- 1 omt wms 890880 Feb 11 10:18 tarfile
> > > -rw-r--r-- 1 omt wms 891360 Feb 11 10:19 tarfile.o
> > >
> > > venus:~/TEST3$ /usr/local/erc32/bin/sparc-rtems-gdb
> > > test.exe
> > > GNU gdb 4.17.gnat.3.11p
> > > Copyright 1998 Free Software Foundation, Inc.
> > > GDB is free software, covered by the GNU General Public License, and you
> > > are
> > > welcome to change it and/or distribute copies of it under certain
> > > conditions.
> > > Type "show copying" to see the conditions.
> > > There is absolutely no warranty for GDB. Type "show warranty" for
> > > details.
> > > This GDB was configured as "--host=i686-pc-linux-gnu
> > > --target=sparc-erc32-elf32"...
> > > (gdb) tar sim -ram 32000 -rom 8000
> > > SIS - SPARC instruction simulator 3.0.5
> > > Bug-reports to Jiri Gaisler ESA/ESTEC (jgais at ws.estec.esa.nl)
> > > serial port A on stdin/stdout
> > > allocated 32768 K RAM memory
> > > allocated 8192 K ROM memory
> > > Connected to the simulator.
> > > (gdb) lo
> > > loading:
> > > section .text at 0x02000000 (106416 bytes)
> > > section .data at 0x02019fb0 (893152 bytes)
> > > section .bss at 0x020f4090 (22124 bytes)(not loaded)
> > > (gdb) bre Init
> > > Breakpoint 1 at 0x2001238: file test.c, line 70.
> > > 77 status = Untar_FromMemory((unsigned char
> > > *)(&_binary_tarfile_start), (unsigned long) &_binary_tarfile_size);
> > > (gdb)
> > > Error during write
> > >
> > > Untar_FromMemory (tar_buf=0x201a0e0 "Klasses.jar", size=890880) at
> > > ../../../../../../rtems-19991203.joel/c/src/libmisc/untar/untar.c:93
> > >
> > > So, the same problem I send you yesterday night by going inside untar:
> > >
> > > 215 for (i=0; i<nblocks; i++)
> > > (gdb)
> > > 217 len = ((sizeToGo < 512L)?(sizeToGo):(512L));
> > > (gdb)
> > > 218 n = fwrite(&tar_buf[ptr], 1, len, fp);
> > > (gdb)
> > > 219 if (n != len)
> > > (gdb)
> > > 224 ptr += 512;
> > >
> > > ++++++++++++++
> > > This is a loop
> > > ++++++++++++++
> > >
> > > (gdb) print nblocks
> > > $118 = 1729
> > > (gdb) print i
> > > $119 = 11
> > >
> > > it needs 5 next for i=i+1
> > > so, I try i=i+1000
> > >
> > > (gdb) next 5000
> > > Error during write
> > >
> > >
> > > I don't know how to solve the problem...
> > >
> > > Thanks for your support,
> > > Oscar
> > > --
> > > Óscar Martínez de la Torre - Office Phone: +31-(0)71.565.5844
> > > Telecommunication Engineer at Universidad Politécnica de Madrid
> > > European Space Agency ESTEC/TOS/EMS Erasmus Building Office Nb212
> > > Keplerlaan,1 - NL 2201 AZ - Noordwijk ZH - The Netherlands
> > > Mail: omt at wm.estec.esa.nl - Personal Mail: omt at etsit.upm.es
> > > /* Init
> > > * This routine is the initialization task for this test program.
> > > */
> > >
> > > #include <bsp.h>
> > >
> > > #define CONFIGURE_TEST_NEEDS_CONSOLE_DRIVER
> > > #define CONFIGURE_TEST_NEEDS_CLOCK_DRIVER
> > > #define CONFIGURE_RTEMS_INIT_TASKS_TABLE
> > >
> > > #define CONFIGURE_EXECUTIVE_RAM_SIZE (512*1024)
> > > #define CONFIGURE_MAXIMUM_SEMAPHORES 20
> > > #define CONFIGURE_MAXIMUM_TASKS 20
> > >
> > > #define CONFIGURE_INIT_TASK_STACK_SIZE (10*1024)
> > > #define CONFIGURE_INIT_TASK_PRIORITY 120
> > > #define CONFIGURE_INIT_TASK_INITIAL_MODES (RTEMS_PREEMPT | \
> > > RTEMS_NO_TIMESLICE | \
> > > RTEMS_NO_ASR | \
> > > RTEMS_INTERRUPT_LEVEL(0))
> > >
> > > #define CONFIGURE_INIT
> > >
> > > rtems_task Init(
> > > rtems_task_argument argument
> > > );
> > >
> > > /* configuration information */
> > >
> > > #include <confdefs.h>
> > >
> > > #include <errno.h>
> > > #include <time.h>
> > >
> > > #include <confdefs.h>
> > > #include <stdio.h>
> > > #include <rtems/rtems_bsdnet.h>
> > > #include <ftpd.h>
> > >
> > > #include <unistd.h>
> > >
> > > #include <rtems/error.h>
> > > #include <rpc/rpc.h>
> > > #include <netinet/in.h>
> > > #include <time.h>
> > >
> > > #include <arpa/inet.h>
> > > #include <sys/socket.h>
> > >
> > > #include <fcntl.h>
> > > #include <rtems/untar.h>
> > >
> > > #define PERMS 0666
> > >
> > >
> > > extern int _binary_tarfile_start;
> > > extern int _binary_tarfile_size;
> > >
> > > rtems_task Init(
> > > rtems_task_argument argument
> > > )
> > >
> > > {
> > >
> > > rtems_status_code status;
> > > int fd;
> > > int fd2;
> > > int ok;
> > > char buffer[5];
> > > char prueba[5]="rtems";
> > >
> > >
> > > printf( "START\n" );
> > >
> > > printf("%s\n",prueba);
> > >
> > > status = Untar_FromMemory((unsigned char *)(&_binary_tarfile_start), (unsigned long) &_binary_tarfile_size);
> > >
> > > fd = open("hola",O_RDWR);
> > > if (fd==-1)
> > > printf("problems open: %i\n",fd);
> > >
> > > while ((ok = read(fd, buffer, 4)) >0)
> > > printf("n:%i leo:%s\n",ok,buffer);
> > >
> > > ok = close(fd);
> > > fd = open("hola",O_RDWR);
> > > if (fd==-1)
> > > printf("problems open: %i\n",fd);
> > >
> > > ok=write(fd,prueba,4);
> > > if (ok==-1)
> > > printf("problems w1: %i\n",ok);
> > >
> > > ok = close(fd);
> > > fd = open("hola",O_RDWR);
> > >
> > > while ((ok = read(fd, buffer, 4)) >0)
> > > printf("2n:%i leo:%s\n",ok,buffer);
> > >
> > > fd2 = creat("adios",PERMS);
> > > if (fd2==-1)
> > > printf("problems creat: %i\n",fd);
> > >
> > > ok=write(fd2,prueba,4);
> > > if (ok==-1)
> > > printf("problems w2: %i\n",ok);
> > >
> > > ok=read(fd2,buffer,4);
> > > if (ok==-1)
> > > printf("problems r2: %i\n",ok);
> > >
> > > printf("%s\n",buffer);
> > > printf("%s\n",prueba);
> > >
> > > exit(0);
> > > }
> > >
> >
> > --
> > janovetz at uiuc.edu | How can it be that mathematics, being after all a
> > University of Illinois | product of human thought independent of experience,
> > | is so admirably adapted to the objects of reality?
> > PP-ASEL | - Albert Einstein
> >
> > Disclaimer: The policies of this University certainly do not reflect my
> > own opinions, objectives, or agenda.
>
> --
> Joel Sherrill, Ph.D. Director of Research & Development
> joel at OARcorp.com On-Line Applications Research
> Ask me about RTEMS: a free RTOS Huntsville AL 35805
> Support Available (256) 722-9985
--
janovetz at uiuc.edu | How can it be that mathematics, being after all a
University of Illinois | product of human thought independent of experience,
| is so admirably adapted to the objects of reality?
PP-ASEL | - Albert Einstein
Disclaimer: The policies of this University certainly do not reflect my
own opinions, objectives, or agenda.
More information about the users
mailing list