cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (CASSANDRA-847) Make the reading half of compactions memory-efficient
Date Sat, 06 Mar 2010 02:03:27 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842152#action_12842152
] 

Jonathan Ellis edited comment on CASSANDRA-847 at 3/6/10 2:03 AM:
------------------------------------------------------------------

> I've always considered the Iterator interface to be super-weak for IO. There is no support
for exception handling, and hasNext() is dangerous (when you begin chaining iterators that
are returning lazy objects like IteratingRow) because it usually involves seeks. 

Statically checked exceptions are a misfeature anyway, so that's a non-issue.

hasNext won't seek any more than a clunkier manual interface.  either way you need to (a)
design your format so you don't have to do random i/o and (b) buffer to avoid seeks when switching
between different objects, and that is what we do.

the old iterator-based compaction for instance never does more seeks than another interface
would given the same buffer sizes.

      was (Author: jbellis):
    > I've always considered the Iterator interface to be super-weak for IO. There is no
support for exception handling, and hasNext() is dangerous (when you begin chaining iterators
that are returning lazy objects like IteratingRow) because it usually involves seeks. 

Typed exceptions are a misfeature anyway, so that's a non-issue.

hasNext won't seek any more than a clunkier manual interface.  either way you need to (a)
design your format so you don't have to do random i/o and (b) buffer to avoid seeks when switching
between different objects, and that is what we do.

the old iterator-based compaction for instance never does more seeks than another interface
would given the same buffer sizes.
  
> Make the reading half of compactions memory-efficient
> -----------------------------------------------------
>
>                 Key: CASSANDRA-847
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-847
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Stu Hood
>            Priority: Critical
>             Fix For: 0.7
>
>         Attachments: 0001-Add-structures-that-were-important-to-the-SSTableSca.patch,
0002-Implement-most-of-the-new-SSTableScanner-interface.patch, 0003-Rename-RowIndexedReader-specific-test.patch,
0004-Improve-Scanner-tests-and-separate-SuperCF-handling-.patch, 0005-Add-Scanner-interface-and-a-Filtered-implementation-.patch,
0006-Add-support-for-compaction-of-super-CFs-and-some-tes.patch
>
>
> This issue is the next on the road to finally fixing CASSANDRA-16. To make compactions
memory efficient, we have to be able to perform the compaction process on the smallest possible
chunks that might intersect and contend one-another, meaning that we need a better abstraction
for reading from SSTables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message