Return-Path: Delivered-To: apmail-db-derby-user-archive@www.apache.org Received: (qmail 8734 invoked from network); 30 Mar 2007 22:39:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 30 Mar 2007 22:39:04 -0000 Received: (qmail 52170 invoked by uid 500); 30 Mar 2007 22:39:10 -0000 Delivered-To: apmail-db-derby-user-archive@db.apache.org Received: (qmail 52141 invoked by uid 500); 30 Mar 2007 22:39:10 -0000 Mailing-List: contact derby-user-help@db.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: List-Post: List-Id: Reply-To: "Derby Discussion" Delivered-To: mailing list derby-user@db.apache.org Received: (qmail 52128 invoked by uid 99); 30 Mar 2007 22:39:10 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Mar 2007 15:39:10 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of raykroeker@gmail.com designates 64.233.184.239 as permitted sender) Received: from [64.233.184.239] (HELO wr-out-0506.google.com) (64.233.184.239) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Mar 2007 15:39:01 -0700 Received: by wr-out-0506.google.com with SMTP id 25so714790wry for ; Fri, 30 Mar 2007 15:38:40 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=m320kSoyMsdFnIh+KhLNkvRC77U0f1FFRfppKxbun6odik997XEHEGctzfy1IBmrQeMgbQk70J4swjfHeZED8XqvSTZi2AS7oIoDox1HMy/75JLecjyUVE2L8wrnUNGnnZGG/IOqHjzQiGMfduigkjrcE4DLnYAAjFrn5Fqf6IE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=tHHNPO3hCUM5Ca8uD8FElDVGZS5x+79rXncHeJvjdn1DcMYfFcEzGyMRcrBlZW18H19ItW8zSuWEaLUT3D6S3uTtxXF3J5xxFtdOWDhQszcY7gzJlRDH7VifmIJ7A4ZSsoFSi8aSX1KJapHQALSUFRI6+vfYKk6ZHz7rPbTqiR4= Received: by 10.115.32.1 with SMTP id k1mr925772waj.1175294319896; Fri, 30 Mar 2007 15:38:39 -0700 (PDT) Received: by 10.114.136.2 with HTTP; Fri, 30 Mar 2007 15:38:39 -0700 (PDT) Message-ID: Date: Fri, 30 Mar 2007 15:38:39 -0700 From: "Raymond Kroeker" Sender: raykroeker@gmail.com To: "Derby Discussion" Subject: Re: Network Client/Server BLOB In-Reply-To: <460AD055.6060504@Sun.COM> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_66659_16659094.1175294319838" References: <4608D5E6.1020705@Sun.com> <460A18F5.9050700@Sun.com> <460AD055.6060504@Sun.COM> X-Google-Sender-Auth: acecc58b2fe31cdf X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_66659_16659094.1175294319838 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Error Description: I was running my test harness code to create a new user. I had a number of poorly coded sql statements causing exceptions, and eventually i came to a point where the derby database server gave up. The only LOB work I was doing was retreiving user vcard info (CLOB) and storing/retreiving application event info (CLOB) both of which are small in size (< 1KB). Stack Trace: 80{770}), Failed Statement is: select P.PRODUCT_ID,P.PRODUCT_NAME from TPSD_PRODUCT P where P.PRODUCT_NAME=? java.lang.OutOfMemoryError: Java heap space at java.io.ByteArrayOutputStream.(ByteArrayOutputStream.java :60) at org.apache.derby.iapi.services.io.AccessibleByteArrayOutputStream.(Unknown Source) at org.apache.derby.iapi.services.classfile.ClassFormatOutput.(Unknown Source) at org.apache.derby.iapi.services.classfile.ClassFormatOutput.(Unknown Source) at org.apache.derby.impl.services.bytecode.CodeChunk.(Unknown Source) at org.apache.derby.impl.services.bytecode.BCClass.newFieldWithAccessors(Unknown Source) at org.apache.derby.impl.sql.compile.ActivationClassBuilder.finishExecuteMethod(Unknown Source) at org.apache.derby.impl.sql.compile.StatementNode.generate(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source) at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source) at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.(Unknown Source) at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.apache.derby.impl.drda.DRDAStatement.prepareStatementJDBC3(Unknown Source) at org.apache.derby.impl.drda.DRDAStatement.prepare(Unknown Source) at org.apache.derby.impl.drda.DRDAStatement.explicitPrepare(Unknown Source) at org.apache.derby.impl.drda.DRDAConnThread.parsePRPSQLSTT(Unknown Source) at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source) at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source) Cleanup action completed On 3/28/07, Kristian Waagan wrote: > > Raymond Kroeker skrev: > > I did run in Java 6 and have a stack, but the stack is without debug > > info. The derby.log is also kinda big so extracting the OOME will > > require a script and a little time. > > > > My confusion stems from the fact that the OOME manifests as a slow > > leak as opposed to direct implementation flaw in the server. I can > > work fine for a while then run into the error as opposed to hitting > > the error consistently with every LOB transaction. > > > > I'll reply again when I have extracted the error. > > Raymond, > > Thank you for taking the time to investigate this :) > > Depending on the what we figure out for this problem, it might be best > to create a Jira issue to track it. We can get back to this later if > necessary. > > If you have a repro, it would be perfect. If you don't, but have a brief > description of the sequence leading to the OOME, I think we have the > required tools in the test-framework to easily write one. If no-one else > feels the itch, I can have a look at it. > > > regards, > -- > Kristian > > > > > On 3/28/07, *Kristian Waagan* > > wrote: > > > > Raymond Kroeker wrote: > > > Hello Kristian, > > > > > > Thank you for your reply. In client/server mode the derby > server > > > currently uses no jvm memory settings, however the derby client > > > application uses -Xms512m -Xmx1024m. > > > > Okay. This is to be expected, as LOBs are currently materialized > > in the > > client when they are retrieved (and in some cases when they are > > sent to > > the server). > > The LOCATOR-work will improve the clients ability to handle large > LOBs > > on the server. > > > > > > > > The same blob is being written to and retreived from the > > database in > > > both the client and server software. I have had successfull > > attempts at > > > storing 5 x 100MB blobs then had an out of memory error within > > the derby > > > network server when retreiving a 12MB blob. > > > > > > 1. My blobs vary in size. Up to 100MB has worked, 12MB had a > > failure > > > in the network server. > > > 2. I insert them using ps.setBinaryStream(1, stream, > > length); and > > > retreive them using rs.getBlob("CONTENT").getBinaryStream(). Both > > > workflows are issued from the database client. > > > > Using the setBinaryStream with a length argument is good, and should > > allow you to stream any Blob to the server. > > > > > 3. The heap size for the derby server is default, the heap > > size of > > > the derb client is -Xms512m -Xmx1024m. > > > > > > I too have had no issue with respect to the embedded client, > > and my > > > out of memory error occured within the network server so I am > > confused a > > > little. > > > > Did you run with Java SE 6 and get a stack trace when the OOME > > happened? > > If so, I'm sure it would be interesting to see where it happened > > for the > > people working on these issues. If you want stack traces with line > > numbers in them, you must download the debug-distribution (from the > > db.apache.org/derby site). > > > > The Derby network server is a component/layer on top of the embedded > > driver. It is possible that even though the embedded driver is > > capable > > of handling Blobs up to 2GB-1, the network server might not be. > > > > > > > > thanks, > > -- > > Kristian > > > > > > > > Raymond > > > > > > On 3/27/07, *Kristian Waagan* < Kristian.Waagan@sun.com > > > > > > >> wrote: > > > > > > Raymond Kroeker wrote: > > > > I have encountered some entries in the mailing list > > suggesting that > > > > large BLOB content is not supported when using the > > > > network client/server. By not supported I mean either > > the client > > > or the > > > > server run out of memory when the content is > > > > sufficiently large. > > > > > > > > My own tests on version 10.2.2.0 > > > > > < http://10.2.2.0> suggest that this is > > > > not the case; however I wanted to know if anyone can > > provide a > > > > confirmation one way or another of this. Thanks. > > > > > > Hello Raymond, > > > > > > From your mail I understand that you are able to use > > sufficiently > > > large > > > BLOBs for you purpose with Derby 10.2.2.0 > > > in > > > client/server mode. > > > Would you mind telling us a little more about your use of > Derby? > > > > > > Quite some work has been done to remove problems with LOB > > handling in > > > Derby, and more work is being done as we speek (implementing > > > LOCATOR-based handling of LOBs). > > > That said, I'm not sure we are exactly where we want to be > > in Derby > > > 10.2.2.0 >>. > > > > > > A few questions; > > > 1) How big are your BLOBs? > > > 2) Do you both insert and retrieve them from/on the client? > > > 3) What are your JVM settings? (heap size) > > > > > > > > > As far as confirmation goes, I have tested embedded earlier > > with 2GB-1 > > > LOBs without out-of-memory problems. I am under the > > impression that > > > stream data still has to be materialized on the client > > before it is sent > > > to the server. > > > Can anyone else confirm this? > > > Will the LOCATOR-work fix the issue? > > > > > > > > > > > > regards, > > > -- > > > Kristian > > > > > > > > > > > -- > > > > --------------------------------------------------------- > > > > Raymond Kroeker > > > > > > > > > > > > > > > -- > > > --------------------------------------------------------- > > > Raymond Kroeker > > > > > > > > > > -- > > --------------------------------------------------------- > > Raymond Kroeker > > -- --------------------------------------------------------- Raymond Kroeker ------=_Part_66659_16659094.1175294319838 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Error Description:
  I was running my test harness code to create a new user.  I had a number of poorly coded sql statements causing exceptions, and eventually i came to a point where the derby database server gave up.

  The only LOB work I was doing was retreiving user vcard info (CLOB) and storing/retreiving application event info (CLOB) both of which are small in size (< 1KB).

Stack Trace:
80{770}), Failed Statement is: select P.PRODUCT_ID,P.PRODUCT_NAME from TPSD_PRODUCT P where P.PRODUCT_NAME=?
java.lang.OutOfMemoryError: Java heap space
        at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:60)
        at org.apache.derby.iapi.services.io.AccessibleByteArrayOutputStream .<init>(Unknown Source)
        at org.apache.derby.iapi.services.classfile.ClassFormatOutput.<init>(Unknown Source)
        at org.apache.derby.iapi.services.classfile.ClassFormatOutput.<init>(Unknown Source)
        at org.apache.derby.impl.services.bytecode.CodeChunk.<init>(Unknown Source)
        at org.apache.derby.impl.services.bytecode.BCClass.newFieldWithAccessors(Unknown Source)
        at org.apache.derby.impl.sql.compile.ActivationClassBuilder.finishExecuteMethod (Unknown Source)
        at org.apache.derby.impl.sql.compile.StatementNode.generate(Unknown Source)
        at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
        at org.apache.derby.impl.sql.GenericStatement.prepare (Unknown Source)
        at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement40 .<init>(Unknown Source)
        at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
        at org.apache.derby.impl.drda.DRDAStatement.prepareStatementJDBC3(Unknown Source)
        at org.apache.derby.impl.drda.DRDAStatement.prepare (Unknown Source)
        at org.apache.derby.impl.drda.DRDAStatement.explicitPrepare(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.parsePRPSQLSTT(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.processCommands (Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
Cleanup action completed


On 3/28/07, Kristian Waagan <Kristian.Waagan@sun.com> wrote:
Raymond Kroeker skrev:
> I did run in Java 6 and have a stack, but the stack is without debug
> info.  The derby.log is also kinda big so extracting the OOME will
> require a script and a little time.
>
> My confusion stems from the fact that the OOME manifests as a slow
> leak as opposed to direct implementation flaw in the server.  I can
> work fine for a while then run into the error as opposed to hitting
> the error consistently with every LOB transaction.
>
> I'll reply again when I have extracted the error.

Raymond,

Thank you for taking the time to investigate this :)

Depending on the what we figure out for this problem, it might be best
to create a Jira issue to track it. We can get back to this later if
necessary.

If you have a repro, it would be perfect. If you don't, but have a brief
description of the sequence leading to the OOME, I think we have the
required tools in the test-framework to easily write one. If no-one else
feels the itch, I can have a look at it.


regards,
--
Kristian

>
> On 3/28/07, *Kristian Waagan* <Kristian.Waagan@sun.com
> <mailto: Kristian.Waagan@sun.com>> wrote:
>
>     Raymond Kroeker wrote:
>     > Hello Kristian,
>     >
>     >   Thank you for your reply.  In client/server mode the derby server
>     > currently uses no jvm memory settings, however the derby client
>     > application uses -Xms512m -Xmx1024m.
>
>     Okay. This is to be expected, as LOBs are currently materialized
>     in the
>     client when they are retrieved (and in some cases when they are
>     sent to
>     the server).
>     The LOCATOR-work will improve the clients ability to handle large LOBs
>     on the server.
>
>     >
>     >   The same blob is being written to and retreived from the
>     database in
>     > both the client and server software.  I have had successfull
>     attempts at
>     > storing 5 x 100MB blobs then had an out of memory error within
>     the derby
>     > network server when retreiving a 12MB blob.
>     >
>     >   1.  My blobs vary in size.  Up to 100MB has worked, 12MB had a
>     failure
>     > in the network server.
>     >   2.  I insert them using  ps.setBinaryStream(1, stream,
>     length); and
>     > retreive them using rs.getBlob("CONTENT").getBinaryStream().  Both
>     > workflows are issued from the database client.
>
>     Using the setBinaryStream with a length argument is good, and should
>     allow you to stream any Blob to the server.
>
>     >   3.  The heap size for the derby server is default, the heap
>     size of
>     > the derb client is -Xms512m -Xmx1024m.
>     >
>     >   I too have had no issue with respect to the embedded client,
>     and my
>     > out of memory error occured within the network server so I am
>     confused a
>     > little.
>
>     Did you run with Java SE 6 and get a stack trace when the OOME
>     happened?
>     If so, I'm sure it would be interesting to see where it happened
>     for the
>     people working on these issues. If you want stack traces with line
>     numbers in them, you must download the debug-distribution (from the
>     db.apache.org/derby <http://db.apache.org/derby> site).
>
>     The Derby network server is a component/layer on top of the embedded
>     driver. It is possible that even though the embedded driver is
>     capable
>     of handling Blobs up to 2GB-1, the network server might not be.
>
>
>
>     thanks,
>     --
>     Kristian
>
>     >
>     > Raymond
>     >
>     > On 3/27/07, *Kristian Waagan* < Kristian.Waagan@sun.com
>     <mailto: Kristian.Waagan@sun.com>
>     > <mailto:Kristian.Waagan@sun.com
>     <mailto: Kristian.Waagan@sun.com>>> wrote:
>     >
>     >     Raymond Kroeker wrote:
>     >      > I have encountered some entries in the mailing list
>     suggesting that
>     >      > large BLOB content is not supported when using the
>     >      > network client/server.  By not supported I mean either
>     the client
>     >     or the
>     >      > server run out of memory when the content is
>     >      > sufficiently large.
>     >      >
>     >      > My own tests on version 10.2.2.0 <http://10.2.2.0>
>     <http://10.2.2.0 >
>     >     < http://10.2.2.0> suggest that this is
>     >      > not the case; however I wanted to know if anyone can
>     provide a
>     >      > confirmation one way or another of this.  Thanks.
>     >
>     >     Hello Raymond,
>     >
>     >      From your mail I understand that you are able to use
>     sufficiently
>     >     large
>     >     BLOBs for you purpose with Derby 10.2.2.0 <http://10.2.2.0>
>     <http://10.2.2.0 <http://10.2.2.0>> in
>     >     client/server mode.
>     >     Would you mind telling us a little more about your use of Derby?
>     >
>     >     Quite some work has been done to remove problems with LOB
>     handling in
>     >     Derby, and more work is being done as we speek (implementing
>     >     LOCATOR-based handling of LOBs).
>     >     That said, I'm not sure we are exactly where we want to be
>     in Derby
>     >     10.2.2.0 < http://10.2.2.0> <http://10.2.2.0 <http://10.2.2.0>>.
>     >
>     >     A few questions;
>     >        1) How big are your BLOBs?
>     >        2) Do you both insert and retrieve them from/on the client?
>     >        3) What are your JVM settings? (heap size)
>     >
>     >
>     >     As far as confirmation goes, I have tested embedded earlier
>     with 2GB-1
>     >     LOBs without out-of-memory problems. I am under the
>     impression that
>     >     stream data still has to be materialized on the client
>     before it is sent
>     >     to the server.
>     >     Can anyone else confirm this?
>     >     Will the LOCATOR-work fix the issue?
>     >
>     >
>     >
>     >     regards,
>     >     --
>     >     Kristian
>     >
>     >      >
>     >      > --
>     >      > ---------------------------------------------------------
>     >      > Raymond Kroeker
>     >
>     >
>     >
>     >
>     > --
>     > ---------------------------------------------------------
>     > Raymond Kroeker
>
>
>
>
> --
> ---------------------------------------------------------
> Raymond Kroeker




--
---------------------------------------------------------
Raymond Kroeker ------=_Part_66659_16659094.1175294319838--