cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5555) Allow sstableloader to handle a larger number of files
Date Thu, 11 Jul 2013 22:55:48 GMT


Jonathan Ellis commented on CASSANDRA-5555:

The logic looks good to me.  Style nits:

- Prefer more concise field names like {{streamingDetails}} to {{endpointToStreamingDetails}}.
 With the type information available, we can see that it's a {{Map<InetAddress, EndpointStreamingDetails>}}
and don't need to redundantly "encode" that in the field name.
- By convention, Java foreach loops have a space before as well as after the colon
- in the {{stream}} method, you replaced a {{for}} block of the form

for ...
    exceptional case

  main case


for ...
    exceptional case
    main case

Generally it's good policy to keep the "main case" less nested, which the if-continue form
lets us do.  This keeps your mental stack less cluttered when you're reasoning about it.

- The {{List<List<Pair<Long, Long>>>}} is probably a bit more hackish than
we want here.  How about adding an sstable reference into the details class so you can associate
them that way, instead of "sstable 0 goes with list entry 0?"  Something like this.  (({{streamingDetails}}
would then become a Multimap.)

    private class SSTableStreamingSections
        public final SSTableReader sstable;

        public final List<Long, Long> sections;
        public final int estimatedKeys;

        private SSTableStreamingSections(SSTableReader sstable, List<Long, Long> sections,
int estimatedKeys)
            this.sstable = sstable;
            this.sections = sections;
            this.estimatedKeys = estimatedKeys;

> Allow sstableloader to handle a larger number of files
> ------------------------------------------------------
>                 Key: CASSANDRA-5555
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core, Tools
>            Reporter: Tyler Hobbs
>            Assignee: Tyler Hobbs
>             Fix For: 1.2.7
>         Attachments: 5555-01.txt, 5555-02.txt, 5555-2.txt, 5555-fix-heap-and-streaming-1.2.patch,
cass_5555_pic_8.png, CASSANDRA-5555.txt, CASSANDRA-5555.txt, CASSANDRA-5555.txt
> With the default heap size, sstableloader will OOM when there are roughly 25k files in
the directory to load.  It's easy to reach this number of files in a single LCS column family.
> By avoiding creating all SSTableReaders up front in SSTableLoader, we should be able
to increase the number of files that sstableloader can handle considerably.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message