Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Date: Thu, 21 Mar 2013 18:15:17 +0000 (UTC)
From: "johannes stadler (JIRA)" <jira@apache.org>
To: derby-dev@db.apache.org
Message-ID: <JIRA.12637173.1363344165059.28645.1363889717991@arcas>
In-Reply-To: <JIRA.12637173.1363344165059@arcas>
References: <JIRA.12637173.1363344165059@arcas>
Subject: [jira] [Commented] (DERBY-6111) OutOfMemoryError using CREATE INDEX
 on large tables
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/DERBY-6111?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1360=
9248#comment-13609248 ]=20

johannes stadler commented on DERBY-6111:
-----------------------------------------


-----------------------------------------------------------------------
Johannes Stadler
--------------- DE ---------------
Ich bin vom 15.03.2013 17:00 - 25.03.2013 07:00 abwesend.
Ich werde Ihre Nachricht nach meiner R=C3=BCckkehr beantworten.
In dringenden F=C3=A4llen wenden Sie sich bitte an unseren Support:
E-Mail: support@foconis.de
Telefon: +49 (0)8541 90390-390
--------------- EN ---------------
I'm out of office 15.03.2013 17:00 - 25.03.2013 07:00.
I will respond to your message after having returned.
In urgent cases please contact our support department:
eMail: support@foconis.de
Phone: +49 (0)8541 90390-390
----------------------------------
---------------------------------------------------------------------
FOCONIS ... mehr Zeit f=C3=BCrs Wesentliche
----------------------------------------------------------------------
Anschrift und Sitz:
FOCONIS AG, Furtgasse 2, 94474 Vilshofen
Telefon: +49 (0) 8541 90390-0
Telefax: +49 (0) 8541 90390-900
Vorstand: Werner Strau=C3=9F, Olaf Pulwey
Aufsichtsrat: Johann M=C3=BChlbauer (Vors.)
Amtsgericht Passau: HRB 6572
USt-ID-Nr: DE 211 388 566
----------------------------------------------------------------------


               =20
> OutOfMemoryError using CREATE INDEX on large tables
> ---------------------------------------------------
>
>                 Key: DERBY-6111
>                 URL: https://issues.apache.org/jira/browse/DERBY-6111
>             Project: Derby
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 10.8.2.2, 10.9.1.0
>         Environment: Windows 7, different JREs (1.4.2_09, 1.5.0_11, 7)
>            Reporter: Johannes Stadler
>            Priority: Critical
>              Labels: CREATE, INDEX, OutOfMemory, OutOfMemoryError
>         Attachments: createIndexOOM.zip, java_pid3236.zip
>
>
> I'm experiencing OutOfMemoryErorrs when performing a simple CREATE INDEX =
command on tables with more than 500,000 rows.=20
> The crashes occured not deterministically in our standard environment usi=
ng 64MByte heap space. But you can easily reproduce the error using the the=
 repro database attached, when running it with 12MByte heap space.
> Just start ij with the -Xmx12M JVM argument, connect to the sample db and=
 execute
> CREATE INDEX IDX_CUSTOMER_NAME ON CUSTOMER (NAME).
> I've done some investigation and i was able to track down the error. It o=
ccurs in SortBuffer.insert(), but not as expected in NodeAllocator.newNode(=
) (there is a handler for the OOE), but already in the call of sortObserver=
.insertDuplicateKey() or .insertNonDuplicateKey() (where the data value des=
criptors are cloned).
> Unfortunately this is not the point to fix it. As i caused the MergeRun (=
that spills the buffer to disk) to happen earlier, it did not significantly=
 lower the memory consumption. Instead it created about 13,000 temp files w=
ith only 1KByte size (because of the many files, performance was inacceptab=
le).
> So i analyzed the heap (using the HeapDumpOnOutOfMemory option) and saw t=
hat it's not the sortbuffer that consumes most of the memory (just few KByt=
es and about 6% of the memory), but the ConcurrentCache. Even though the ma=
xSize of the ConcurrentCache was set 1000, the cache contained about 2,500 =
elements. I've also attached the heapdump.
> If i'm understanding the concept right, the cache elements are added with=
out regarding the maxSize and there's a worker thread that runs on low prio=
, that shrinks the cache from time to time to 10% of its size.
> I think in this particular case, where memory is getting low, it would be=
 a better idea to have the cache cleared synchronously and provide more spa=
ce to the sortBuffer. Maybe that could be done in the ClockPolicy.insertEnt=
ry() in case, that the current size is increasing the max size by 50%. I'm =
not already very familiar with the code, so i failed to do so.
> I hope you got all the information you need, if you require any further i=
nformation, please let me know.
> Greetings
> Johannes Stadler

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira