cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonas Borgström (Updated) (JIRA) <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-3466) Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2
Date Thu, 10 Nov 2011 12:55:52 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonas Borgström updated CASSANDRA-3466:
---------------------------------------

    Attachment: CASSANDRA-3466-2.tar.gz

I've unfortunately managed to reproduce this AssertionError even with the CASSANDRA-3446 patches
and the HintedHandOffManager.java modification.

I used the same two node setup as before initialized with:

{code}
create keyspace foo with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
and strategy_options = {replication_factor:2};
use foo;
create column family bar with comparator=UTF8Type and key_validation_class=UTF8Type;
{code}

# Shutdown node2 and wait until it's detected as down
trigger hints on node1 (using random values):
{code}
use foo;
set bar[0][0]='1';
set bar[2][3]='1';
set bar[3][3]='1';
{code}
Verify that `list HintsColumnFamily` looks OK
# Start node2 and wait for hints delivery
Verify that `list HintsColumnFamily` is empty after hints delivery

After repeating step 1-2 a few times (sometimes once is enough) "HintsColumnFamily" will become
"corrupt" after hints delivery instead of empty. And if node1 is restarted after that it will
trigger the AssertionError.

The attached file contains DEBUG-log files, data directories and conf-directories from one
test where HintsColumnFamily ended up looking like this:
{code}
[default@system] list HintsColumnFamily;
Using default limit of 100
-------------------
RowKey: 1b226da5af66854850abdcc6ab4ce9c6
=> (super_column=1f2860e00b9811e10000fe8ebeead9ff,
     (column=6b6579, value=33, timestamp=1320928378352, ttl=864000))

1 Row Returned.
Elapsed time: 6 msec(s).
{code}

                
> Hinted handoff not working after rolling upgrade from 0.8.7 to 1.0.2
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-3466
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3466
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Jonas Borgström
>            Assignee: Brandon Williams
>              Labels: hintedhandoff
>             Fix For: 1.0.3
>
>         Attachments: CASSANDRA-3466-2.tar.gz, CASSANDRA-3466.tar.gz
>
>
> While testing rolling upgrades from 0.8.7 to 1.0.2 on a test cluster I've noticed that
hinted hand-off didn't always work properly. Hints generated on an upgraded node does not
seem to be delivered to other newly upgraded nodes once they rejoin the ring. They only way
I've found to get a node to deliver its hints is to restart it.
> Here's some steps to reproduce this issue:
> 1. Install cassandra 0.8.7 on node1 and node2 using default settings.
> 2. Create keyspace foo with {replication_factor: 2}. Create column family bar
> 3. Shutdown node2 
> 4. Insert data into bar and verify that HintsColumnFamily on node2 contains hints
> 5. Start node2 and verify that hinted handoff is performed and HintsColumnFamily becomes
empty again.
> 6. Upgrade and restart node1
> 7. Shutdown node2 
> 8. Insert data into bar and verify that HintsColumnFamily on node2 contains hints
> 9. Upgrade and start node2
> 10. Notice that hinted handoff is *not* performed when "node2" comes back. (Only if node1
is restarted)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message