db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig L Russell <Craig.Russ...@Sun.COM>
Subject Re: Cloudscape Database Size limitation
Date Wed, 23 Nov 2005 23:30:04 GMT

On Nov 23, 2005, at 6:49 AM, Michael Segel wrote:

> I'm a little skeptical whenever I see a question like this.
> There are always limits.
> And while you could get up to 100 GB, I wouldn't recommend it  
> performance
> wise as Derby is currently written.

Well, if I understand it correctly, IBM have the resources to  
actually set up and test databases of that size, and I'm happy that  
they have, and will do so. It should be a learning experience for  
them, and I expect that they will share with the community whatever  
they learn about Derby, warts and all.
> I think the important point isn't to find a theoretical limit, but  
> to choose
> the right database for the right job.

I can't disagree!
> ...
> Going over a certain point, you're going to want to choose a  
> database that
> has a better storage method than using the file system.
> (And I had better clarify...)
> I'm talking about having the data stored in either raw or cooked  
> chunks. Not
> As separate files within a directory. (More akin to Informix's  
> Standard
> Engine.)

Well, my experience does not lead me to believe that a database  
engine will necessarily be better at managing persistent storage via  
more primitive i/o, compared to letting the operating system manage  
the data. There are so many parameters that affect disk performance  
that "better storage method than... the file system" trivializes the  

Just off the top, how memory is used to store the user's record (row)  
data, how that memory is mapped to the operating system's i/o  
buffers, whether logs or data pages hold "the truth", how the data to  
be written is organized for multiple users, what operating system  
calls are used to write and/or verify the data, whether the writes  
are ordered by physical location on disk, whether the disk controller  
is buffered, the size of the buffers, and what form of verification  
that the write has succeeded the engine requests.

Few of these issues have anything to do with whether the disk space  
is "raw", "cooked", or in a file system.
> Its not like I haven't been playing with databases since the 1980s?
> (eeew! Now I know I'm getting old.) ;-)

The raw/cooked/files discussion reminds me a lot of the assembler/C/C+ 
+ arguments over the years. A tiny percentage of programmers these  
days have time to understand the architecture of a modern CPU chip  
and how to optimize a program based on it. We leave that to the  
language runtime implementers.

And few of us are interested enough and need to learn the ins and  
outs of writing raw data to partitions. We leave these details to the  
operating system gurus. Give me a nice file abstraction and high  
performance, direct i/o buffers and let the experts play in their own  

> -G

Craig Russell
Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
408 276-5638 mailto:Craig.Russell@sun.com
P.S. A good JDO? O, Gasp!

View raw message