cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua McKenzie (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-3668) Parallel streaming for sstableloader
Date Tue, 04 Mar 2014 22:29:43 GMT


Joshua McKenzie commented on CASSANDRA-3668:

With streaming being overhauled in 2.0 it looks like I'll need to revisit the core implementation
on this.

Got the following from Yuki:
"In streaming 2.0, transferring/receiving files to one destination is managed by StreamSession
To parallelize streaming, I think the easiest way is to make multiple StreamSession per host,
instead of one per host that what we do right now.
So given set of SSTable files in bulk loader, divide those up to groups equal to number of
parallel streaming, and assign each group to multiple StreamSessions of the same destination.

One thing to be careful in this approach is, you need to add one more id to streaming messages
in order to distinguish messages to the same destination.
This also changes serialization format of the protocol so need to update streaming protocol
version defined in StreamMessage."

> Parallel streaming for sstableloader
> ------------------------------------
>                 Key: CASSANDRA-3668
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: API
>            Reporter: Manish Zope
>            Assignee: Joshua McKenzie
>            Priority: Minor
>              Labels: streaming
>             Fix For: 2.1 beta2
>         Attachments: 3668-1.1-v2.txt, 3668-1.1.txt, 3688-reply_before_closing_writer.txt,
sstable-loader performance.txt
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> One of my colleague had reported the bug regarding the degraded performance of the sstable
generator and sstable loader.
> ISSUE :- 
> As stated in above issue generator performance is rectified but performance of the sstableloader
is still an issue.
> 3589 is marked as duplicate of 3618.Both issues shows resolved status.But the problem
with sstableloader still exists.
> So opening other issue so that sstbleloader problem should not go unnoticed.
> FYI : We have tested the generator part with the patch given in 3589.Its Working fine.
> Please let us know if you guys require further inputs from our side.

This message was sent by Atlassian JIRA

View raw message