cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lu Ming (JIRA)" <>
Subject [jira] Commented: (CASSANDRA-1169) AES makes Streaming unhappy
Date Sat, 12 Jun 2010 11:00:15 GMT


Lu Ming commented on CASSANDRA-1169:

After I applied the above patch,
StreamOutManager.waitForStreamCompletion()  return immediately and StreamOut.transferSSTables
do Not  wait for its streaming tasks to finish

27938- INFO [STREAM-STAGE:1] 2010-06-12 17:08:48,810 (line 132) Sending a stream
initiate message to /
27939: INFO [STREAM-STAGE:1] 2010-06-12 17:08:48,810 (line 137) Waiting for
transfer to / to complete
27940- INFO [STREAM-STAGE:1] 2010-06-12 17:08:48,810 (line 141) Done with transfer
to /
27941- INFO [AE-SERVICE-STAGE:1] 2010-06-12 17:08:48,811 (line 641)
Finished streaming repair to / for (GroupDataStore,Group)
27982- INFO [STREAM-STAGE:1] 2010-06-12 17:19:22,066 (line 132) Sending a stream
initiate message to / ...
27983: INFO [STREAM-STAGE:1] 2010-06-12 17:19:22,066 (line 137) Waiting for
transfer to / to complete
27984- INFO [STREAM-STAGE:1] 2010-06-12 17:19:22,066 (line 141) Done with transfer
to /
27985- INFO [AE-SERVICE-STAGE:1] 2010-06-12 17:19:22,067 (line 641)
Finished streaming repair to / for (GroupChat,Topic)

> AES makes Streaming unhappy
> ---------------------------
>                 Key: CASSANDRA-1169
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Gary Dusbabek
>            Priority: Critical
>             Fix For: 0.6.3, 0.7
>         Attachments: 1169.txt, aes.txt
> Streaming service assumes there will only be one stream from S to T at a time for any
nodes S and T.  For the original purpose of node movement, this was a reasonable assumption
(any node T can only perform one move at a time) but AES throws off streaming tasks much more
frequently than that given the right conditions, which will de-sync the fragile file ordering
that Streaming assumes (that T knows which files S is going to send, in what order).  Eventually
T is expecting file F1 but S sends a smaller file F2, leading to an infinite loop on T while
it waits for F1 to finish, and T waits for S to acknowledge F2, which it never will.
> For 0.6 maybe the best solution is for AES to manually wait for one of its streaming
tasks to finish, before it allows itself to create another.  For 0.7 it would be nice to make
Streaming more robust.  The whole 4-stage-ack process seems very fragile, and poking around
in parent objects via inetaddress keys makes reasoning about small pieces impossible b/c of
encapsulation violations.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message