cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-208) jvm crashes intermittently during compaction
Date Fri, 29 May 2009 23:26:59 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714635#action_12714635
] 

Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------

blockindexes_ keeps the indexes in memory as the flush or compaction is performed, so they
can be dumped at the end of the SSTable file.

The old code on code.google dumped each index at the end of its block which avoided this problem
(only one block's index would ever be in memory at once).  I'm not sure why FB changed this
behavior.  The only reason I can think of is that reading the index later won't require as
many seeks, but this seems like a bad optimization to make since index reading is done once
per sstable.

I couldn't tell at first why this should cause OOM -- in both the google and apache versions
we have indexMetadataMap_ which contains the same block index info, right?  Wrong, the difference
is that iMM only contains the first entry from each block index.  So much less memory is used
there.

I guess we should switch back to the old, interleaved block indexing method.

> jvm crashes intermittently during compaction
> --------------------------------------------
>
>                 Key: CASSANDRA-208
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-208
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: trunk
>         Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5 
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
>            Reporter: Jiansheng Huang
>
> jvm crashes intermittently during compaction. Our test data set is not that big, less
than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory (exceeding the max
heap size).
> The excessive memory usage during compaction is caused by the maintenance of blockIndexes_
in SSTable. this blockIndexes_ was only introduced to the apache version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message