cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonas Borgström (Commented) (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3466) Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2
Date Tue, 08 Nov 2011 07:40:51 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146124#comment-13146124
] 

Jonas Borgström commented on CASSANDRA-3466:
--------------------------------------------

> Do you observe the same behavior if both nodes start life as 1.x?

No. Hinted handoff seems to work well if I either start with 1.0.2 nodes or restart all nodes
once all nodes have been upgraded to 1.0.2. Probably something in the gossip state confusing
hinted hand-offs.

> Is node2 down long enough for node1 failure detector to notice? (i.e. shows as DOWN in
nodetool ring)

Yes. This from the node1 log:
{code}
Node2 is shut down
  INFO 19:39:08,331 InetAddress /127.0.0.2 is now dead.
Hint is triggered using set bar[x][x]='x'
  INFO 19:40:52,126 Node /127.0.0.2 has restarted, now UP
  INFO 19:40:52,127 InetAddress /127.0.0.2 is now UP
  INFO 19:40:52,127 Node /127.0.0.2 state jump to normal
Nothing happens, hint stays in HintsColumnFamily forever
{code}

Calling deliverHints() using JMX also does not seem to work. Restarting node1 will though...

                
> Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-3466
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3466
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.2
>            Reporter: Jonas Borgström
>              Labels: hintedhandoff
>
> While testing rolling upgrades from 0.8.7 to 1.0.2 on a test cluster I've noticed that
hinted hand-off didn't always work properly. Hints generated on an upgraded node does not
seem to be delivered to other newly upgraded nodes once they rejoin the ring. They only way
I've found to get a node to deliver its hints is to restart it.
> Here's some steps to reproduce this issue:
> 1. Install cassandra 0.8.7 on node1 and node2 using default settings.
> 2. Create keyspace foo with {replication_factor: 2}. Create column family bar
> 3. Shutdown node2 
> 4. Insert data into bar and verify that HintsColumnFamily on node2 contains hints
> 5. Start node2 and verify that hinted handoff is performed and HintsColumnFamily becomes
empty again.
> 6. Upgrade and restart node1
> 7. Shutdown node2 
> 8. Insert data into bar and verify that HintsColumnFamily on node2 contains hints
> 9. Upgrade and start node2
> 10. Notice that hinted handoff is *not* performed when "node2" comes back. (Only if node1
is restarted)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message