db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <mse...@segel.com>
Subject Re: Cloudscape Database Size limitation
Date Thu, 24 Nov 2005 20:53:36 GMT
On Wednesday 23 November 2005 17:30, Craig L Russell wrote:
> Hi,
> > ...
> > Going over a certain point, you're going to want to choose a
> > database that
> > has a better storage method than using the file system.
> > (And I had better clarify...)
> >
> > I'm talking about having the data stored in either raw or cooked
> > chunks. Not
> > As separate files within a directory. (More akin to Informix's
> > Standard
> > Engine.)
> Well, my experience does not lead me to believe that a database
> engine will necessarily be better at managing persistent storage via
> more primitive i/o, compared to letting the operating system manage
> the data. There are so many parameters that affect disk performance
> that "better storage method than... the file system" trivializes the
> issue.
I wasn't sure on how to respond to this.

I've had a couple of different conversations with some of the "boffins" who 
were part of the ORDMBS crew at Informix.  I think we all shared the same 
opinion that one of the short comings of those who come from a Java OO 
background and not a RDBMS background tended to miss out on the significance 
that a RDBMS is more than a persistant object store.

OODBMS don't scale. (there's a reason for this...)

I mean if you go back to the concept of a bean being able to store itself, it 
makes sense from an OO point of view. But what happens when that bean is 
composed of a master/detail relationship? 

Use a simple invoice as an example. You're going to have to store the header 
information separately from the detail information.

This is probably why Sun really hasn't developed their own RDMBS... ;-)

You have to really think beyond "persistance of storage".

> Just off the top, how memory is used to store the user's record (row)
> data, how that memory is mapped to the operating system's i/o
> buffers, whether logs or data pages hold "the truth", how the data to
> be written is organized for multiple users, what operating system
> calls are used to write and/or verify the data, whether the writes
> are ordered by physical location on disk, whether the disk controller
> is buffered, the size of the buffers, and what form of verification
> that the write has succeeded the engine requests.
> Few of these issues have anything to do with whether the disk space
> is "raw", "cooked", or in a file system.
Again, doesn't scale well and has a lot of unnecessary overhead.
> ...
> > Its not like I haven't been playing with databases since the 1980s?
> > (eeew! Now I know I'm getting old.) ;-)
> The raw/cooked/files discussion reminds me a lot of the assembler/C/C+
> + arguments over the years. A tiny percentage of programmers these
> days have time to understand the architecture of a modern CPU chip
> and how to optimize a program based on it. We leave that to the
> language runtime implementers.
The assembler vs C argument was over long ago. When the C compilers were able 
to generate efficient code, the need to do hand coded assembler went away. 
The point being that the incremental benefits gained by hand coding assembler 
could not justify the additional expense and loss of portability.

Note that this does not excuse the software engineer from his/her 
responsibility of understanding the language that they are working with and 
in their use of good coding techniques.  

> And few of us are interested enough and need to learn the ins and
> outs of writing raw data to partitions. We leave these details to the
> operating system gurus. Give me a nice file abstraction and high
> performance, direct i/o buffers and let the experts play in their own
> domains.
You know if you ever get a chance to work at the OS level, you should take it. 
Its a blast. And it teaches you that you have to really think about what 
you're doing. It will definitely improve your overall design and coding 

> Craig
> > -G
> Craig Russell
> Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
> 408 276-5638 mailto:Craig.Russell@sun.com
> P.S. A good JDO? O, Gasp!

You have to understand that its one thing to write apps that call a database, 
and its another to actually write the database software itself.

You actually are in some instances getting down to the OS level.

A good example of this is Informix's use of threads.... 

Michael Segel
312 952- 8175 [M]

View raw message