cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tania S Engel (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-14006) Migration task failed completes bootstrap but it didn't stream any data.
Date Thu, 09 Nov 2017 19:39:00 GMT
Tania S Engel created CASSANDRA-14006:
-----------------------------------------

             Summary: Migration task failed completes bootstrap but it didn't stream any data.
                 Key: CASSANDRA-14006
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14006
             Project: Cassandra
          Issue Type: Bug
          Components: Streaming and Messaging
            Reporter: Tania S Engel


When joining just one node with very little data we often get "Migration task failed to complete"
per https://github.com/apache/cassandra/commit/ae315b5ec944571342146867c51b2ceb50f3845e
 
We increased the timeout on *MIGRATION_TASK_WAIT_IN_SECONDS *to 15 minutes thinking there
was some sort of auto retry mechanism in the underlying messaging. However all that does is
increase the time to failure. When these migration tasks fail, the bootstrap is marked complete
but it clearly wasn't complete because usage of the data results in a cassandra.db.*UnknownColumnFamilyException*.
Also, it is evident in the logs that no data was streamed from the seed node to the newly
bootstrapping node. We have had numerous tests showing that if a migration task times out,
the node exits joining mode, the bootstrap logs complete, but it hasn't streamed any data
and the only course of action seems to be a Cassandra restart. Our replication factor is set
such that the bootstrapping node needs to get all the data. If we were to leave the Cassandra
node running, would it eventually send another migration task and stream the necessary data?
 
On closer inspection of the code, it seems that the *MigrationTask.java* runMayThrow sends
the migration request message using *sendRR*, which is a fire and forget. So, if the callback
is not hit, it can leave you in a state where the CountdownLatch.countDown() is never invoked.
So, I suppose that is the point of the timeout when waiting for the latch. But wouldn't it
be better to resend the migration task? I certainly haven't learned all the messaging service
but it seems that dropping a packet here and there could cause bootstrap to succeed in this
misleading way. Would it make sense for the MigrationTask runMayThrow to create a IAsyncCallbackWithFailure
for the callback and implement the OnFailure to also CountdownLatch.countDown() and generate
another migration task? Or perhaps allow users of Cassandra to configure something like a
MIGRATION_TASK_RETRY_ATTEMPTS?
 
When the MigrationTask does fail to complete, we see the log 3 times. Is this the resend of
the same migration task …which is just schema version exchange? In which case if all 3 failed
it means all attempts failed to reach the seed endpoint or the response failed to reach the
bootstrapping endpoint. Are we correct in assuming this is a network error and there are no
scenarios where the seed node would ignore the migration task from the bootstrapping node?





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message