cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-2946) HintedHandoff fails with could not reach schema agreement
Date Tue, 26 Jul 2011 19:22:09 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071298#comment-13071298
] 

Brandon Williams commented on CASSANDRA-2946:
---------------------------------------------

+1, what happened was node A never updated it's gossip state, but the other node learned of
its schema version by RPC.  When node B restarted, it no longer knew the schema version for
A, and tried to get it from gossip where it was still old.

bq. if it does we can also change the HH code to use DD.
I don't know about that, having this as a check to make sure this doesn't happen again is
a far easier thing to catch than other "gossip is out of date" problems we might encounter.

> HintedHandoff fails with could not reach schema agreement
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-2946
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2946
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.1
>            Reporter: Richard Low
>            Assignee: Jonathan Ellis
>             Fix For: 0.8.3
>
>         Attachments: 2946.txt
>
>
> To reproduce, have two nodes A and B.
> 1. On node A, create a keyspace with replication factor 1 and add a column family
> 2. Ensure node B has created the keyspace and column family
> 3. Take down node B
> 4. Insert some keys to A at CL.ANY, ensuring some keys should be written to B
> 5. Bring up node B
> 6. When hints are delivered, I get the error:
> ERROR [HintedHandoff:1] 2011-07-25 17:19:14,729 AbstractCassandraDaemon.java (line 139)
Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement
with /10.2.129.9 in 60000ms
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with /10.2.129.9
in 60000ms
>         at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:290)
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:301)
>         at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
>         at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:394)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         ... 3 more
> If I use DatabaseDescriptor.getDefsVersion() instead of gossiper.getEndpointStateForEndpoint(FBUtilities.getLocalAddress()).getApplicationState(ApplicationState.SCHEMA)
then the error goes away, and the hints are correctly delivered.
> This may be the same issue as Aaron saw here: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/ApplicationState-Schema-has-drifted-from-DatabaseDescriptor-td6006576.html,
and may be related to CASSANDRA-2083.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message