db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DERBY-5416) SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
Date Wed, 11 Dec 2013 10:58:09 GMT

    [ https://issues.apache.org/jira/browse/DERBY-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845286#comment-13845286
] 

ASF subversion and git services commented on DERBY-5416:
--------------------------------------------------------

Commit 1550103 from [~knutanders] in branch 'code/trunk'
[ https://svn.apache.org/r1550103 ]

DERBY-5416: SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the
heap is full at call time and then gets mostly garbage collected later
on.

Improve the accuracy of the code that estimates the memory requirement
of the sort buffer. When it detects that the current memory usage is
lower than the initial memory usage, it now records the current usage
and uses that value instead of the initial memory usage in future
calculations.

This compensates to some degree, but not fully, for the skew in the
estimates due to garbage collection happening after the initial memory
usage. The memory requirement will not be as badly underestimated, and
the likelihood of OutOfMemoryErrors is reduced.

There is no regression test case for this bug, since the only
reliable, reproducible test case that we currently have, needs too
much time, disk space and memory to be included in the regression test
suite.

> SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and
then gets mostly garbage collected later on
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: DERBY-5416
>                 URL: https://issues.apache.org/jira/browse/DERBY-5416
>             Project: Derby
>          Issue Type: Bug
>          Components: Store
>    Affects Versions: 10.6.2.1, 10.7.1.1, 10.8.1.2
>            Reporter: Ramin Baradari
>            Assignee: Knut Anders Hatlen
>            Priority: Critical
>              Labels: derby_triage10_9
>         Attachments: compress_test_5416.patch, d5416-1a.diff, lowmem-test.diff
>
>
> When compressing a table with an index that is larger than the maximum heap size and
therefore cannot be hold in memory as a whole an OutOfMemoryError can occur. 
> For this to happen the heap usage must be close to the maximum heap size at the start
of the index recreation and then while the entries are sorted a garbage collection run must
clean out most of the heap. This can happen because a concurrent process releases a huge chunk
of memory or just because the buffer of a previous table compression has not yet been garbage
collected. 
> The internally used heuristics to guess when more memory can be used for the merge inserter
estimates that more memory is available and then the sort buffer gets doubled. The buffer
size gets doubled until the heap usage is back to the level when the merge inserter was first
initialized or when the OOM occurs.
> The problem lies in MergeInsert.insert(...). The check if the buffer can be doubled contains
the expression "estimatedMemoryUsed < 0" where estimatedMemoryUsed is the difference in
current heap usage and heap usage at initialization. Unfortunately, in the aforementioned
scenario this will be true until the heap usage will reach close to maximum heap size before
doubling the buffer size will be stopped.
> I've tested it with 10.6.2.1, 10.7.1.1 and 10.8.1.2 but the actual bug most likely exists
in prior versions too.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message