Return-Path: Delivered-To: apmail-db-derby-user-archive@www.apache.org Received: (qmail 88209 invoked from network); 23 Nov 2005 23:30:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 23 Nov 2005 23:30:35 -0000 Received: (qmail 29403 invoked by uid 500); 23 Nov 2005 23:30:34 -0000 Delivered-To: apmail-db-derby-user-archive@db.apache.org Received: (qmail 29380 invoked by uid 500); 23 Nov 2005 23:30:34 -0000 Mailing-List: contact derby-user-help@db.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: List-Post: List-Id: Reply-To: "Derby Discussion" Delivered-To: mailing list derby-user@db.apache.org Received: (qmail 29369 invoked by uid 99); 23 Nov 2005 23:30:34 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Nov 2005 15:30:34 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=HTML_MESSAGE,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [192.18.98.34] (HELO brmea-mail-3.sun.com) (192.18.98.34) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Nov 2005 15:32:05 -0800 Received: from fe-amer-09.sun.com ([192.18.108.183]) by brmea-mail-3.sun.com (8.12.10/8.12.9) with ESMTP id jANNUC3F004122 for ; Wed, 23 Nov 2005 16:30:12 -0700 (MST) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-4.02 (built Sep 9 2005)) id <0IQF00601L19OQ00@mail-amer.sun.com> (original mail from Craig.Russell@Sun.COM) for derby-user@db.apache.org; Wed, 23 Nov 2005 16:30:12 -0700 (MST) Received: from [192.168.0.10] ([24.6.172.77]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-4.02 (built Sep 9 2005)) with ESMTPSA id <0IQF004XRLA7CJLI@mail-amer.sun.com> for derby-user@db.apache.org; Wed, 23 Nov 2005 16:30:12 -0700 (MST) Date: Wed, 23 Nov 2005 15:30:04 -0800 From: Craig L Russell Subject: Re: Cloudscape Database Size limitation In-reply-to: <20051123155445.874CE36FB1@WebRack01.Segel.com> Sender: Craig.Russell@Sun.COM To: Derby Discussion Message-id: <6D6E4C4F-3BFD-4A92-AA96-0C6EA6BC35D9@sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.746.2) Content-type: multipart/alternative; boundary="Boundary_(ID_xT1jI5V9/yiwRzs4DbCX0A)" References: <20051123155445.874CE36FB1@WebRack01.Segel.com> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N --Boundary_(ID_xT1jI5V9/yiwRzs4DbCX0A) Content-type: text/plain; format=flowed; delsp=yes; charset=US-ASCII Content-transfer-encoding: 7BIT Hi, On Nov 23, 2005, at 6:49 AM, Michael Segel wrote: > I'm a little skeptical whenever I see a question like this. > There are always limits. > > And while you could get up to 100 GB, I wouldn't recommend it > performance > wise as Derby is currently written. Well, if I understand it correctly, IBM have the resources to actually set up and test databases of that size, and I'm happy that they have, and will do so. It should be a learning experience for them, and I expect that they will share with the community whatever they learn about Derby, warts and all. > > I think the important point isn't to find a theoretical limit, but > to choose > the right database for the right job. I can't disagree! > ... > Going over a certain point, you're going to want to choose a > database that > has a better storage method than using the file system. > (And I had better clarify...) > > I'm talking about having the data stored in either raw or cooked > chunks. Not > As separate files within a directory. (More akin to Informix's > Standard > Engine.) Well, my experience does not lead me to believe that a database engine will necessarily be better at managing persistent storage via more primitive i/o, compared to letting the operating system manage the data. There are so many parameters that affect disk performance that "better storage method than... the file system" trivializes the issue. Just off the top, how memory is used to store the user's record (row) data, how that memory is mapped to the operating system's i/o buffers, whether logs or data pages hold "the truth", how the data to be written is organized for multiple users, what operating system calls are used to write and/or verify the data, whether the writes are ordered by physical location on disk, whether the disk controller is buffered, the size of the buffers, and what form of verification that the write has succeeded the engine requests. Few of these issues have anything to do with whether the disk space is "raw", "cooked", or in a file system. > ... > Its not like I haven't been playing with databases since the 1980s? > (eeew! Now I know I'm getting old.) ;-) The raw/cooked/files discussion reminds me a lot of the assembler/C/C+ + arguments over the years. A tiny percentage of programmers these days have time to understand the architecture of a modern CPU chip and how to optimize a program based on it. We leave that to the language runtime implementers. And few of us are interested enough and need to learn the ins and outs of writing raw data to partitions. We leave these details to the operating system gurus. Give me a nice file abstraction and high performance, direct i/o buffers and let the experts play in their own domains. Craig > > -G > > > Craig Russell Architect, Sun Java Enterprise System http://java.sun.com/products/jdo 408 276-5638 mailto:Craig.Russell@sun.com P.S. A good JDO? O, Gasp! --Boundary_(ID_xT1jI5V9/yiwRzs4DbCX0A) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: QUOTED-PRINTABLE Hi,

On Nov = 23, 2005, at 6:49 AM, Michael Segel wrote:

I'm a little skeptical when= ever I see a question like this.
There are = always limits.

=
And while you could get up to 100 GB, I wouldn't= recommend it performance
wise as Derby is = currently written.

Well, if I understand it correctly, IBM have the res= ources to actually set up and test databases of that size, and I'm ha= ppy that they have, and will do so. It should be a learning experienc= e for them, and I expect that they will share with the community what= ever they learn about Derby, warts and all.=A0

I think the important point isn't to find a theoretical li= mit, but to choose
the right database for t= he right job.=A0

I can't d= isagree!
...
Going over a certain point, you're going to want to cho= ose a database that
has a better storage me= thod than using the file system. =A0
(And I had better clarify...)<= /DIV>

I'm talking about having the data stored in either raw or cooke= d chunks. Not
As separate files within a di= rectory. (More akin to Informix's Standard
= Engine.)

Well, my experience does not lead me to believe that a databas= e engine will necessarily be better at managing persistent storage vi= a more primitive i/o, compared to letting the operating system manage= the data. There are so many parameters that affect disk performance = that "better storage method than... the file system" trivializes the = issue.

Jus= t off the top, how memory is used to store the user's record (row) da= ta, how that memory is mapped to the operating system's i/o buffers, = whether logs or data pages hold "the truth", how the data to be writt= en is organized for multiple users, what operating system calls are u= sed to write and/or verify the data, whether the writes are ordered b= y physical location on disk, whether the disk controller is buffered,= the size of the buffers, and what form of verification that the writ= e has succeeded the engine requests.

Few of these issues have anything to do wi= th whether the disk space is "raw", "cooked", or in a file system.

...
Its not like = I haven't been playing with databases since the 1980s?
(eeew! Now I know I'm getting old.) ;-)

The raw/cooked/file= s discussion reminds me a lot of the assembler/C/C++ arguments over t= he years. A tiny percentage of programmers these days have time to un= derstand the architecture of a modern CPU chip and how to optimize a = program based on it. We leave that to the language runtime implemente= rs.=A0

And= few of us are interested enough and need to learn the ins and outs o= f writing raw data to partitions. We leave these details to the opera= ting system gurus. Give me a nice file abstraction and high performan= ce, direct i/o buffers and let the experts play in their own domains.=

Craig
=

=
-G




<= DIV>

Craig Russell

Architect, Sun Jav= a Enterprise System http= ://java.sun.com/products/jdo

408 276-5638 mailto:Craig.Russell@sun.com

P.S. A good JDO? O, Gasp!


--Boundary_(ID_xT1jI5V9/yiwRzs4DbCX0A)--