cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Schuller (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-3882) avoid distributed deadlock in migration stage
Date Thu, 10 May 2012 03:34:52 GMT


Peter Schuller updated CASSANDRA-3882:

    Attachment: CASSANDRA-3882-hack.txt

Attaching a pure hack that works around this on large clusters (schema migrations are essentially
impossible/useless on large clusters without it).

It is not a clean solution, but it was very fast to hack together, should be very safe, and
solves our burning problem. It simply spreads out migration tasks slowly over time so that
the probability of triggering deadlocks becomes vastly smaller. (We're also running with a
hack to make the timeout on migration messages be 500 ms.)

You'll tend to see schema converging very quickly, and then see a flurry of secondly memtable
flushes as it processes rectification in the background, even though they don't actually "do"
anything to the schema.

> avoid distributed deadlock in migration stage
> ---------------------------------------------
>                 Key: CASSANDRA-3882
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Peter Schuller
>            Assignee: Peter Schuller
>         Attachments: CASSANDRA-3882-hack.txt
> This is follow-up work for the remainders of CASSANDRA-3832 which was only a partial
fix. The deadlock in the migration stage needs to be fixed, as it can cause bootstrap (at
least) to take potentially a very very long time to complete, and might also cause a lack
of schema propagation until otherwise "poked".

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message