cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5222) OOM Exception during repair session with LeveledCompactionStrategy
Date Wed, 13 Feb 2013 09:10:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577420#comment-13577420
] 

Sylvain Lebresne commented on CASSANDRA-5222:
---------------------------------------------

bq. I noticed that Validator.add already asserts that the row is in the range being validated,
so we must be doing a range check somewhere.

The validation is intrinsically limited to a range, so ValidationCompactionIterable make sure
we only scan said range through
{noformat}
cfs.getCompactionStrategy().getScanners(sstables, range)
{noformat}
However, looking at it, it's not very efficient because currently we end up creating an SSTableBoundedScanner
object even if the range is not covered at all by the sstable. And while said SSTableBoundedScanner
will be correct in that it will be exhausted right away, the way the code works, we still
open the data file and so we still allocate the reader buffer(s). So, probably a separate
issue but we should optimize that nonetheless. And while looking at that, I realized that
LeveledScanner poorly compute the getLengthInBytes info as it assumes we cover the whole ring
(we will still end up at 100% because getCurrentPosition() will also return a value assuming
the whole ring, but it's still a poor indicator of the actual work we do/have to do).


                
> OOM Exception during repair session with LeveledCompactionStrategy
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-5222
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5222
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>         Environment: 3Gb Heap(12Gb per node RAM)
> 36 nodes, 0.9 Tb of data per node, Leveled compaction strategy, SSTable size =100Mb
>            Reporter: Ivan Sobolev
>            Assignee: Jonathan Ellis
>             Fix For: 1.1.11
>
>         Attachments: 5222.txt, chunks.json, sstablescanner.png
>
>
> 1.8 Gb of heap is consumed with 12k SSTableBoundedScanner * 140kbytes

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message