db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "johannes stadler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DERBY-6111) OutOfMemoryError using CREATE INDEX on large tables
Date Wed, 11 Dec 2013 14:19:08 GMT

    [ https://issues.apache.org/jira/browse/DERBY-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845425#comment-13845425
] 

johannes stadler commented on DERBY-6111:
-----------------------------------------




-----------------------------------------------------------------------
Johannes Stadler
--------------- DE ---------------
Ich bin vom 29.11.2013 17:00 - 02.01.2014 07:00 abwesend.

Ich werde Ihre Nachricht nach meiner Rückkehr beantworten.
In dringenden Fällen wenden Sie sich bitte an unseren Support:
E-Mail: support@foconis.de
Telefon: +49 (0)8541 90390-390

--------------- EN ---------------
I'm out of office 29.11.2013 17:00 - 02.01.2014 07:00.

I will respond to your message after having returned.
In urgent cases please contact our support department:
eMail: support@foconis.de
Phone: +49 (0)8541 90390-390

----------------------------------

---------------------------------------------------------------------
FOCONIS ... mehr Zeit fürs Wesentliche
----------------------------------------------------------------------
Anschrift und Sitz:
FOCONIS AG, Furtgasse 2, 94474 Vilshofen
Telefon: +49 (0) 8541 90390-0
Telefax: +49 (0) 8541 90390-900
Vorstand: Werner Strauß, Olaf Pulwey
Aufsichtsrat: Johann Mühlbauer (Vors.)
Amtsgericht Passau: HRB 6572
USt-ID-Nr: DE 211 388 566
----------------------------------------------------------------------



> OutOfMemoryError using CREATE INDEX on large tables
> ---------------------------------------------------
>
>                 Key: DERBY-6111
>                 URL: https://issues.apache.org/jira/browse/DERBY-6111
>             Project: Derby
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 10.8.2.2, 10.9.1.0, 10.10.1.1
>         Environment: Windows 7, different JREs (1.4.2_09, 1.5.0_11, 7)
>            Reporter: Johannes Stadler
>            Priority: Critical
>              Labels: CREATE, INDEX, OutOfMemory, OutOfMemoryError, derby_triage10_11
>         Attachments: createIndexOOM.zip, java_pid3236.zip
>
>
> I'm experiencing OutOfMemoryErorrs when performing a simple CREATE INDEX command on tables
with more than 500,000 rows. 
> The crashes occured not deterministically in our standard environment using 64MByte heap
space. But you can easily reproduce the error using the the repro database attached, when
running it with 12MByte heap space.
> Just start ij with the -Xmx12M JVM argument, connect to the sample db and execute
> CREATE INDEX IDX_CUSTOMER_NAME ON CUSTOMER (NAME).
> I've done some investigation and i was able to track down the error. It occurs in SortBuffer.insert(),
but not as expected in NodeAllocator.newNode() (there is a handler for the OOE), but already
in the call of sortObserver.insertDuplicateKey() or .insertNonDuplicateKey() (where the data
value descriptors are cloned).
> Unfortunately this is not the point to fix it. As i caused the MergeRun (that spills
the buffer to disk) to happen earlier, it did not significantly lower the memory consumption.
Instead it created about 13,000 temp files with only 1KByte size (because of the many files,
performance was inacceptable).
> So i analyzed the heap (using the HeapDumpOnOutOfMemory option) and saw that it's not
the sortbuffer that consumes most of the memory (just few KBytes and about 6% of the memory),
but the ConcurrentCache. Even though the maxSize of the ConcurrentCache was set 1000, the
cache contained about 2,500 elements. I've also attached the heapdump.
> If i'm understanding the concept right, the cache elements are added without regarding
the maxSize and there's a worker thread that runs on low prio, that shrinks the cache from
time to time to 10% of its size.
> I think in this particular case, where memory is getting low, it would be a better idea
to have the cache cleared synchronously and provide more space to the sortBuffer. Maybe that
could be done in the ClockPolicy.insertEntry() in case, that the current size is increasing
the max size by 50%. I'm not already very familiar with the code, so i failed to do so.
> I hope you got all the information you need, if you require any further information,
please let me know.
> Greetings
> Johannes Stadler



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message