cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cathy Daw (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (CASSANDRA-3290) Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created in 0.7.9
Date Sat, 01 Oct 2011 05:34:45 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118677#comment-13118677
] 

Cathy Daw edited comment on CASSANDRA-3290 at 10/1/11 5:33 AM:
---------------------------------------------------------------

I see the same issue upgrading from 0.8.2, but now I see the following exception. In this
case, one node was drained and upgraded to 1.0 and the other two nodes were still at 0.8.2.

This is a pretty straight forward procedure using a vanilla install and the out of the box
stress scripts.

{code}

ERROR 05:27:33,751 Fatal exception in thread Thread[HintedHandoff:1,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with
/50.57.111.189 in 60000ms
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with /50.57.111.189
in 60000ms
	at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:220)
	at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:231)
	at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
	at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:333)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more
ERROR 05:27:33,753 Fatal exception in thread Thread[HintedHandoff:1,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with
/50.57.111.189 in 60000ms
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with /50.57.111.189
in 60000ms
	at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:220)
	at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:231)
	at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
	at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:333)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more
{code}

*Node Tool output for 1 node in 1.0.0 and the other two at 0.8.2*
{code}
cathy1:~/apache-cassandra-1.0.0-rc1-src/bin# ./nodetool -h cathy1 ring
Address         DC          Rack        Status State   Load            Owns    Token     
                                 
                                                                               113427455640312821154458202477256070484
    
50.57.111.111   datacenter1 rack1       Up     Normal  181.56 MB       33.33%  0         
                                 
50.57.111.126   datacenter1 rack1       Up     Normal  178.84 MB       33.33%  56713727820156410577229101238628035242
     
50.57.111.189   datacenter1 rack1       Up     Normal  180.2 MB        33.33%  113427455640312821154458202477256070484
    
{code}
                
      was (Author: cdaw):
    I see the same issue upgrading from 0.8.2
                  
> Upgrade from 0.7.9 to 1.0, the java stress tool gives "error reading key" on data created
in 0.7.9
> --------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-3290
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3290
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Cathy Daw
>            Assignee: Brandon Williams
>             Fix For: 1.0.0
>
>
> *TEST CASE*
> * Install three node cluster with C* 0.7.9 with node1 as the seed
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> * Run `nodetool -h node1 drain`
> * Upgrade binaries and edit cassandra.yaml for node1.  
> * Startup C* 1.0 on node
> * Run the 0.7.9 stress tool against node2,node3: ./stress -d node2,node3 -o insert
> *ISSUE*
> * Run the 0.7.9 stress tool to read data: ./stress -d node2 -o read
> {code}
> cathy3:~/apache-cassandra-0.7.9-src/contrib/stress/bin# ./stress -d cathy3 -o read
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> Operation [17] retried 10 times - error reading key 0521977 
> Operation [9] retried 10 times - error reading key 0595365 
> Operation [15] retried 10 times - error reading key 0513247 
> {code}
> *Additional notes*
> * There are no errors in the log files
> * After upgrading the full ring, and running READ from both the old and new versions
of stress tool, the error still exists
> * Once all nodes are upgraded, if I run INSERT followed by READ, the new reads are fine

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message