commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thorbjørn Ravn Andersen <>
Subject Re: flexible byte-stream storage? [vfs] [cache] ??
Date Wed, 21 Feb 2007 18:16:54 GMT
Paul Libbrecht skrev  den 13-02-2007 15:36:
> hello List,
> maybe this should be directed to some other list... I am in the quest 
> of a library to store, key-based, byte-streams of very varying sizes 
> in large quantities. I need to read and write, multithreaded, and I 
> have big and small byte-streams, from 50 bytes till 3 mega-bytes.
> Files work, of course, but there are very many small fragments there, 
> like 100 bytes or so. This is the solution I am using, I have a 
> directory of about 27000 files right now... berk.
Perhaps you could elaborate a bit on which key you use to look up the 
byte stream and how these are generated.

If the filenames are evenly distributed, you could have a 
K/KE/KEYOFDATA.xml two-level hierarchy to easen the load off the filesystem.

You would most likely also look into using an appropriate filesystem for 
the task.  Linux e.g. has several, with different properties.

So, do tell more :)


View raw message