cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuki Morishita (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4297) Use java NIO as much as possible when streaming compressed SSTables
Date Mon, 04 Jun 2012 17:38:23 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288717#comment-13288717
] 

Yuki Morishita commented on CASSANDRA-4297:
-------------------------------------------

CompressedDIS is DataInputStream version of CompressedRandomAccessReader. It reads compressed
chunks directly from stream and provides decompressed data  while reading from stream.  CRC
check is also performed after decompressing chunk based on crc_chance setting in compression
option, which is default to 1.0 or 100%, as done in CRAR.
                
> Use java NIO as much as possible when streaming compressed SSTables
> -------------------------------------------------------------------
>
>                 Key: CASSANDRA-4297
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4297
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Yuki Morishita
>            Assignee: Yuki Morishita
>            Priority: Minor
>              Labels: streaming
>             Fix For: 1.2
>
>
> Back in 0.8, streaming uses java NIO (FileChannel#transferTo/transferFrom) to perform
zero copy file transfer between nodes. Since 1.0, in order to add new features like sstable
compression and internode encryption we had to switch to java IO Input/OutputStreams. What
we currently do to transfer compressed SSTable is, in source node, 1) decompress chunk in
SSTable, 2) compress using LZF for network, and in destination node, 3) decompress using LZF
as reading from socket, 4) compress for SSTable on disk.
> Now, 1.1 comes out with SSTable compression turned on by default. It is reasonable to
transfer compressed file as is using NIO instead of decompress/compress in source node.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message