Re: [SLUG] 100k files in one directory

From: Logan (logan1304@home.net)
Date: Wed Dec 12 2001 - 20:37:45 EST


Oh, I forgot to mention I have 512Mb RAM and a PIII 750 in the server.
On Wed, 2001-12-12 at 19:56, Logan wrote:
> I am using ext2. How many files can reiserFS handle in one directory?
> Or does it matter since it writes a journal?
>
> On Wed, 2001-12-12 at 12:11, Derek Glidden wrote:
> > On Tue, 2001-12-11 at 19:46, Logan wrote:
> > > Various folks in my office dumped just shy of 100K files to a linux
> > > file server I made for back ups. But, they dumped them all into one
> > > directory. There are so many files there that ls times out.
> > > I think the cure would be to recompile a kernel, editing
> > > /usr/src/linux/include/linux/fs.h to change the NR_FILE variable from
> > > 8192 to 8192*10 or even 8192*100. I think this would allow things like
> > > ls and rm to work. Or am I barking up the wrong tree?
> > > Has anyone else had any experience like this? I know I can also cure
> > > it by making more subdirectories and force them to put there files in
> > > the subdirectories. But chiding from the Windoze users stirs me to try
> > > to let them leave their crap all in one big heap.
> >
> > What filesystem are you using?
> >
> > If ext2, there are some patches floating around that dramatically
> > increase the speed of file I/O against directories with lots of files.
> > You'll have to either ask on lkml or check Kernel-Traffic to find the
> > source of them, though, because I don't remember where they are.
> >
> > ReiserFS really should be able to handle it without much trouble.
> >
> > XFS and JFS I'm not so sure of, although I use XFS virtually everywhere
> > nowadays and have no problems with it, and have a server at home with
> > some "junk" directories that get thousands of files dumped into them and
> > have never had issues.
> >
> > Also, how much RAM and what CPU are in that server? Adding more RAM
> > will help the system cache filesystem ops better.
> >
> > If your Windows people are hassling you, you can just come back with
> > "So, what do you do if NTFS falls over on something? What other
> > filesystems can you use? FAT32?" and then laugh.
> >
> > --
> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> > #!/usr/bin/perl -w
> > $_='while(read+STDIN,$_,2048){$a=29;$b=73;$c=142;$t=255;@t=map
> > {$_%16or$t^=$c^=($m=(11,10,116,100,11,122,20,100)[$_/16%8])&110;
> > $t^=(72,@z=(64,72,$a^=12*($_%16-2?0:$m&17)),$b^=$_%64?12:0,@z)
> > [$_%8]}(16..271);if((@a=unx"C*",$_)[20]&48){$h=5;$_=unxb24,join
> > "",@b=map{xB8,unxb8,chr($_^$a[--$h+84])}@ARGV;s/...$/1$&/;$d=
> > unxV,xb25,$_;$e=256|(ord$b[4])<<9|ord$b[3];$d=$d>>8^($f=$t&($d
> > >>12^$d>>4^$d^$d/8))<<17,$e=$e>>8^($t&($g=($q=$e>>14&7^$e)^$q*
> > 8^$q<<6))<<9,$_=$t[$_]^(($h>>=8)+=$f+(~$g&$t))for@a[128..$#a]}
> > print+x"C*",@a}';s/x/pack+/g;eval
> >
> > usage: qrpff 153 2 8 105 225 < /mnt/dvd/VOB_FILENAME \
> > | extract_mpeg2 | mpeg2dec -
> >
> > http://www.cs.cmu.edu/~dst/DeCSS/Gallery/
> > http://www.eff.org/ http://www.anti-dmca.org/
> --
> 7:50pm up 6 days, 12:33, 1 user, load average: 0.79, 0.41, 0.42
> Frisbeetarianism, n.: The belief that when you die, your soul goes up
> the on roof and gets stuck.
>

-- 
8:35pm up 6 days, 13:18, 1 user, load average: 0.00, 0.03, 0.05
"It's bad luck to be superstitious." -- Andrew W. Mathis



This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 20:11:41 EDT