cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Corentin Chary (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-4663) Streaming sends one file at a time serially.
Date Thu, 05 Jan 2017 06:38:59 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798727#comment-15798727
] 

Corentin Chary edited comment on CASSANDRA-4663 at 1/5/17 6:38 AM:
-------------------------------------------------------------------

The thing is that as far as I understand StreamSession#getOrCreateNextSession() will create
multiple sessions:

{code}
INFO  [main] 2017-01-04 16:42:41,003 StorageService.java:1438 - JOINING: Starting to bootstrap...
INFO  [main] 2017-01-04 16:42:41,532 StreamResultFuture.java:90 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Executing streaming plan for Bootstrap
INFO  [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,538 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.115
INFO  [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,541 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#0] Beginning stream session with /10.50.4.115
INFO  [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,562 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.115 is complete
INFO  [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,574 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.115
INFO  [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,576 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#1] Beginning stream session with /10.50.4.115
INFO  [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,594 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.115 is complete
INFO  [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,601 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.115
INFO  [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,602 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#2] Beginning stream session with /10.50.4.115
INFO  [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,607 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.115 is complete
INFO  [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,609 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.126
INFO  [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,610 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#0] Beginning stream session with /10.50.4.126
INFO  [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,676 StreamResultFuture.java:173 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b ID#0] Prepare completed. Receiving 7 files(1.181GiB),
sending 0 files(0.000KiB)
INFO  [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,679 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.126
INFO  [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,682 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#1] Beginning stream session with /10.50.4.126
INFO  [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,718 StreamResultFuture.java:173 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b ID#1] Prepare completed. Receiving 8 files(772.249MiB),
sending 0 files(0.000KiB)
INFO  [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,719 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.126
INFO  [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,720 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#2] Beginning stream session with /10.50.4.126
INFO  [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,726 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.126 is complete
{code}

Because of fetchAsync() iterates on keyspaces to call requestRanges() this will spread the
keyspaces across multiple sessions (it would be even better to spread the ranges).

Since the bottleneck is the CPU bound STREAM-IN thread (IncomingMessageHandler) it will improve
the performances by creating multiple of these threads.

With three keyspaces and three connections I was able to double the throughput and hopefully
that is not just placebo effect.




was (Author: iksaif):
The thing is that as far as I understand {code}StreamSession#getOrCreateNextSession(){code}
will create multiple sessions:

{code}
INFO  [main] 2017-01-04 16:42:41,003 StorageService.java:1438 - JOINING: Starting to bootstrap...
INFO  [main] 2017-01-04 16:42:41,532 StreamResultFuture.java:90 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Executing streaming plan for Bootstrap
INFO  [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,538 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.115
INFO  [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,541 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#0] Beginning stream session with /10.50.4.115
INFO  [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,562 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.115 is complete
INFO  [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,574 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.115
INFO  [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,576 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#1] Beginning stream session with /10.50.4.115
INFO  [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,594 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.115 is complete
INFO  [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,601 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.115
INFO  [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,602 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#2] Beginning stream session with /10.50.4.115
INFO  [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,607 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.115 is complete
INFO  [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,609 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.126
INFO  [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,610 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#0] Beginning stream session with /10.50.4.126
INFO  [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,676 StreamResultFuture.java:173 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b ID#0] Prepare completed. Receiving 7 files(1.181GiB),
sending 0 files(0.000KiB)
INFO  [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,679 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.126
INFO  [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,682 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#1] Beginning stream session with /10.50.4.126
INFO  [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,718 StreamResultFuture.java:173 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b ID#1] Prepare completed. Receiving 8 files(772.249MiB),
sending 0 files(0.000KiB)
INFO  [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,719 StreamSession.java:266 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Starting streaming to /10.50.4.126
INFO  [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,720 StreamCoordinator.java:264 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b, ID#2] Beginning stream session with /10.50.4.126
INFO  [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,726 StreamResultFuture.java:187 -
[Stream #cef1c0e0-d29c-11e6-b978-d922683f145b] Session with /10.50.4.126 is complete
{code}

And because of {code}fetchAsync(){code} iterates on keyspaces to call {code}requestRanges(){code}
this will spread the keyspace accross multiple sessions (it would be even better to spread
the ranges).

And because the bottleneck is the STREAM-IN thread (IncomingMessageHandler) it will improve
the performances.

With three keyspaces and three connections I was able to double the throughput and hopefully
that is not just placebo effect.



> Streaming sends one file at a time serially. 
> ---------------------------------------------
>
>                 Key: CASSANDRA-4663
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4663
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: sankalp kohli
>            Priority: Minor
>             Fix For: 3.x
>
>         Attachments: 0001-streaming-add-a-way-to-configure-the-number-of-conne.patch
>
>
> This is not fast enough when someone is using SSD and may be 10G link. We should try
to create multiple connections and send multiple files in parallel. 
> Current approach under utilize the link(even 1G).
> This change will improve the bootstrapping time of a node. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message