cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan McGuire (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-5461) LOCAL_QUORUM reads/writes don't throw UnavailableException if local quorum not present
Date Sun, 14 Apr 2013 16:58:15 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631347#comment-13631347
] 

Ryan McGuire edited comment on CASSANDRA-5461 at 4/14/13 4:57 PM:
------------------------------------------------------------------

I haven't been able to reproduce this with just stress and cassandra, on 1.1.9, nor 1.2. Here's
what I've tried:

* Create a 4 node 1.1.9 cluster in a single datacenter:
{code}
$ ccm create -v 1.1.9 5641-1.1.9
$ ccm populate -n 4
$ ccm start
$ ccm node1 ring
Note: Ownership information does not include topology, please specify a keyspace. 
Address         DC          Rack        Status State   Load            Owns              
 Token                                       
                                                                                         
 127605887595351923798765477786913079296     
127.0.0.1       datacenter1 rack1       Up     Normal  11.15 KB        25.00%            
 0                                           
127.0.0.2       datacenter1 rack1       Up     Normal  6.73 KB         25.00%            
 42535295865117307932921825928971026432      
127.0.0.3       datacenter1 rack1       Up     Normal  13.52 KB        25.00%            
 85070591730234615865843651857942052864      
127.0.0.4       datacenter1 rack1       Up     Normal  9.09 KB         25.00%            
 127605887595351923798765477786913079296
{code}

* Write to the cluster with a network topology and consistency level of 4:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -e QUORUM
{code}

* Read from the cluster to make sure it's available:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -o read -e QUORUM
{code}

* Shut down nodes 2-4, breaking quorum and local_quorum consistency:
{code}
$ ccm node2 stop
$ ccm node3 stop
$ ccm node4 stop
{code}

This breaks quorum consistency:
{code}
$ ./cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -o read -e QUORUM
total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
Operation [20] retried 10 times - error reading key 052627 ((UnavailableException))
[...]
0,0,0,0.0,0.0,0.0,0
FAILURE
{code}

And it breaks local_quorum consistency:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -o read -e LOCAL_QUORUM
total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
Operation [12] retried 10 times - error reading key 059221 ((UnavailableException))
[...]
0,0,0,0.0,0.0,0.0,0
FAILURE
{code}

I'm thinking there must be some other problem with hadoop going on here? Maybe it's just hiding
the error for some reason?
                
      was (Author: enigmacurry):
    I haven't been able to reproduce this with just stress and cassandra, on 1.1.9, nor 1.2.
Here's what I've tried:

* Create a 4 node 1.1.9 cluster in a single datacenter:
{code}
$ ccm create -v 1.1.9 5641-1.1.9
$ ccm populate -n 4
$ ccm start
$ ccm node1 ring
Note: Ownership information does not include topology, please specify a keyspace. 
Address         DC          Rack        Status State   Load            Owns              
 Token                                       
                                                                                         
 127605887595351923798765477786913079296     
127.0.0.1       datacenter1 rack1       Up     Normal  11.15 KB        25.00%            
 0                                           
127.0.0.2       datacenter1 rack1       Up     Normal  6.73 KB         25.00%            
 42535295865117307932921825928971026432      
127.0.0.3       datacenter1 rack1       Up     Normal  13.52 KB        25.00%            
 85070591730234615865843651857942052864      
127.0.0.4       datacenter1 rack1       Up     Normal  9.09 KB         25.00%            
 127605887595351923798765477786913079296
{code}

* Write to the cluster with a network topology and consistency level of 4:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -e QUORUM
{code}

* Read from the cluster to make sure it's available:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -o read -e QUORUM
{code}

* Shut down nodes 2-4, breaking quorum and local_quorum consistency:
{code}
$ ccm node2 stop
$ ccm node3 stop
$ ccm node4 stop
{code}

This breaks quorum consistency:
{code}
$ ./cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -o read -e QUORUM
total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
Operation [20] retried 10 times - error reading key 052627 ((UnavailableException))
[...]
0,0,0,0.0,0.0,0.0,0
FAILURE
{code}

And it breaks local_quorum consistency:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O datacenter1:4
-n 100000 -o read -e LOCAL_QUORUM
total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
Operation [12] retried 10 times - error reading key 059221 ((UnavailableException))
[...]
0,0,0,0.0,0.0,0.0,0
FAILURE
{code}

I'm thinking there must be some other probken with hadoop going on here? Maybe it's just hiding
the error for some reason?
                  
> LOCAL_QUORUM reads/writes don't throw UnavailableException if local quorum not present
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5461
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5461
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.9
>         Environment: Cassandra 1.1.9.3, DSE 3.0.1, Ubuntu Linux 12.10
>            Reporter: Piotr Kołaczkowski
>            Assignee: Ryan McGuire
>
> {noformat}
> automaton@ip-10-171-69-238:~$ dsetool ring cfs
> Address         DC          Rack        Workload        Status State   Load         
  Effective-Ownership Token                                       
>                                                                                     
                      127605887595351923798765477786913079296     
> 10.171.69.238   Analytics   rack1       Analytics(JT)   Up     Normal  151.19 KB    
  100.00%             0                                           
> 10.171.147.24   Analytics   rack1       Analytics(TT)   Down   Down    103.79 KB    
  100.00%             42535295865117307932921825928971026432      
> 10.172.86.251   Analytics   rack1       Analytics(TT)   Down   Down    145.03 KB    
  100.00%             85070591730234615865843651857942052864      
> 10.174.23.34    Analytics   rack1       Analytics(TT)   Down   Down    158.88 KB    
  100.00%             127605887595351923798765477786913079296
> {noformat}
> {noformat}
> Keyspace: cfs:
>   Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
>   Durable Writes: true
>     Options: [analytics:4]
> {noformat}
> Reading from CFS with LOCAL_QUORUM:
> {noformat}
> automaton@ip-10-171-69-238:~$ dse hadoop fs -lsr /
> 13/04/12 16:18:23 INFO cfs.CassandraFileSystem: CassandraFileSystem.uri : cfs://ip-10-171-69-238.us-west-1.compute.internal/
> 13/04/12 16:18:23 INFO cfs.CassandraFileSystem: Default block size: 67108864
> 13/04/12 16:18:24 INFO cfs.CassandraFileSystemThriftStore: Consistency level for reads
from cfs: LOCAL_QUORUM
> 13/04/12 16:18:24 INFO cfs.CassandraFileSystemThriftStore: Consistency level for writes
into cfs: LOCAL_QUORUM
> 13/04/12 16:18:24 INFO cfs.CassandraFileSystemRules: Successfully loaded path rules for:
cfs
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:42 /tf1
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:49 /tf10
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:43 /tf2
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:43 /tf3
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:43 /tf4
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:46 /tf5
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:46 /tf6
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:47 /tf7
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:47 /tf8
> -rwxrwxrwx   1 automaton automaton         10 2013-04-12 15:49 /tf9
> drwxrwxrwx   - automaton automaton          0 2013-04-12 15:36 /tmp
> drwxrwxrwx   - automaton automaton          0 2013-04-12 15:36 /tmp/hadoop-automaton
> drwxrwxrwx   - automaton automaton          0 2013-04-12 15:36 /tmp/hadoop-automaton/mapred
> drwx------   - automaton automaton          0 2013-04-12 15:36 /tmp/hadoop-automaton/mapred/system
> -rwxrwxrwx   1 automaton automaton          4 2013-04-12 15:36 /tmp/hadoop-automaton/mapred/system/jobtracker.info
> {noformat}
> Now exactly the same with QUORUM:
> {noformat}
> automaton@ip-10-171-69-238:~$ dse hadoop fs -lsr /
> 13/04/12 16:19:21 INFO cfs.CassandraFileSystem: CassandraFileSystem.uri : cfs://ip-10-171-69-238.us-west-1.compute.internal/
> 13/04/12 16:19:21 INFO cfs.CassandraFileSystem: Default block size: 67108864
> 13/04/12 16:19:22 INFO cfs.CassandraFileSystemThriftStore: Consistency level for reads
from cfs: QUORUM
> 13/04/12 16:19:22 INFO cfs.CassandraFileSystemThriftStore: Consistency level for writes
into cfs: QUORUM
> 13/04/12 16:19:22 INFO cfs.CassandraFileSystemRules: Successfully loaded path rules for:
cfs
> lsr: UnavailableException()
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message