Re: [SLUG] 100k files in one directory

From: Derek Glidden (dglidden@illusionary.com)
Date: Fri Dec 14 2001 - 12:54:04 EST


On Thu, 2001-12-13 at 20:05, Ed Centanni wrote:
> This is a a user education issue not a technical one.

I agree, but I've also had many many occasions where the response to
that statement has been, "Yes, but it's a management decision to fix it
with technology, not education." *sigh*

FWIW - I made a spare partition on my server (with LVM) and formatted it
ext2, reiserfs and XFS and made 100K files on it. Making the files took
forever on ext2, but doing an "ls" on the directory was quite fast. (of
course, it hadn't had time to fragment its directory inodes so much...)
I think if I had waited for it to "rm -rf" all those files, it might
have taken until the heat-death of the Universe though... :) ReiserFS
and XFS were both snappy at making and "ls'ing" the files, but XFS still
does "rm" synchronously, so that took forever as well. ReiserFS on the
other hand, is remarkably fast at doing "rm" on 100K files since it
tends to buffer meta-data type operations as well.

I would highly suggest converting that partition to ReiserFS. (Or XFS,
but you'll have to have a system supporting the 2.4 series of kernels
for that, while there are ReiserFS patches for 2.2.) If nothing else, at
least you'll have journalling protection for those 100K+, assumably
"important" work files - and remember, e2fsck takes proportionately long
as the size of the filesystem plus the number of files on that
filesystem.

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#!/usr/bin/perl -w
$_='while(read+STDIN,$_,2048){$a=29;$b=73;$c=142;$t=255;@t=map
{$_%16or$t^=$c^=($m=(11,10,116,100,11,122,20,100)[$_/16%8])&110;
$t^=(72,@z=(64,72,$a^=12*($_%16-2?0:$m&17)),$b^=$_%64?12:0,@z)
[$_%8]}(16..271);if((@a=unx"C*",$_)[20]&48){$h=5;$_=unxb24,join
"",@b=map{xB8,unxb8,chr($_^$a[--$h+84])}@ARGV;s/...$/1$&/;$d=
unxV,xb25,$_;$e=256|(ord$b[4])<<9|ord$b[3];$d=$d>>8^($f=$t&($d
>>12^$d>>4^$d^$d/8))<<17,$e=$e>>8^($t&($g=($q=$e>>14&7^$e)^$q*
8^$q<<6))<<9,$_=$t[$_]^(($h>>=8)+=$f+(~$g&$t))for@a[128..$#a]}
print+x"C*",@a}';s/x/pack+/g;eval 

usage: qrpff 153 2 8 105 225 < /mnt/dvd/VOB_FILENAME \ | extract_mpeg2 | mpeg2dec -

http://www.cs.cmu.edu/~dst/DeCSS/Gallery/ http://www.eff.org/ http://www.anti-dmca.org/



This archive was generated by hypermail 2.1.3 : Fri Aug 01 2014 - 20:14:56 EDT