cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-5555) Allow sstableloader to handle a larger number of files
Date Wed, 19 Jun 2013 05:47:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687629#comment-13687629
] 

Jonathan Ellis edited comment on CASSANDRA-5555 at 6/19/13 5:46 AM:
--------------------------------------------------------------------

To clarify: Alex was still reproducing OOM, but his initial heap dump showed that IndexSummary
was the primary culprit.  Once we started running the code from current 1.2, it still OOMs
but the culprit is now BufferedSegmentedFile.  (Looks like our reader pooling is biting us
in the butt.)
                
      was (Author: jbellis):
    To clarify: Alex was still reproducing OOM, but his initial heap dump showed that IndexSummary
was the primary culprit.  Once we started running the code from current 1.2, it still OOMs
but the culprit is now BufferedSegmentedFile.
                  
> Allow sstableloader to handle a larger number of files
> ------------------------------------------------------
>
>                 Key: CASSANDRA-5555
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5555
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core, Tools
>            Reporter: Tyler Hobbs
>            Assignee: Jonathan Ellis
>             Fix For: 1.2.6
>
>         Attachments: 5555-01.txt, 5555-02.txt, cass_5555_pic_8.png, CASSANDRA-5555.txt,
CASSANDRA-5555.txt
>
>
> With the default heap size, sstableloader will OOM when there are roughly 25k files in
the directory to load.  It's easy to reach this number of files in a single LCS column family.
> By avoiding creating all SSTableReaders up front in SSTableLoader, we should be able
to increase the number of files that sstableloader can handle considerably.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message