cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Zarutin (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5555) Allow sstableloader to handle a larger number of files
Date Wed, 07 Aug 2013 10:28:48 GMT


Alex Zarutin commented on CASSANDRA-5555:

Verified the fix on latest Cassandra 1.2 with about 42 thousands of SSTable files using the
"how to reproduce" steps already attached. 
During the processing the files, I did not see big change in memory consumption and CPU utilization.
Running the process with 256M of the memory as Java -Xmx, the real usage stayed around 140M
during the entire process. 
Suggest to resolve this issue, unless we really want to test it on thousand of huge files
- much bigger that 1M that we used in this test.
> Allow sstableloader to handle a larger number of files
> ------------------------------------------------------
>                 Key: CASSANDRA-5555
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core, Tools
>            Reporter: Tyler Hobbs
>            Assignee: Tyler Hobbs
>             Fix For: 1.2.7, 2.0 beta 2
>         Attachments: 5555-01.txt, 5555-02.txt, 5555-2.txt, 5555-fix-heap-and-streaming-1.2.patch,
5555-fix-heap-and-streaming-1.2-v2.patch, 5555-fix-heap-and-streaming-trunk.patch, cass_5555_pic_8.png,
CASSANDRA-5555.txt, CASSANDRA-5555.txt, CASSANDRA-5555.txt
> With the default heap size, sstableloader will OOM when there are roughly 25k files in
the directory to load.  It's easy to reach this number of files in a single LCS column family.
> By avoiding creating all SSTableReaders up front in SSTableLoader, we should be able
to increase the number of files that sstableloader can handle considerably.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message