cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
Date Wed, 18 Feb 2015 02:19:11 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wei Zhu updated CASSANDRA-8819:
-------------------------------
    Description: 
We have two DC3, each with 7 nodes.
Here is the keyspace setup:

 create keyspace test
 with placement_strategy = 'NetworkTopologyStrategy'
 and strategy_options = {DC2 : 3, DC1 : 3}
 and durable_writes = true;

We brought down two nodes in DC2 for maintenance. We only write to DC1 using local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to client since local_quorum should succeed.

Here are the output from nodetool status

Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns   Host ID                               Rack
UN  10.2.0.1  10.92 GB   256     7.9%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC206
UN  10.2.0.2   6.17 GB    256     8.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC106
UN  10.2.0.3  6.63 GB    256     7.3%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC107
DL  10.2.0.4  1.54 GB    256     7.7%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC107
UN  10.2.0.5  6.02 GB    256     6.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC106
UJ  10.2.0.6   3.68 GB    256     ?      XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC205
UN  10.2.0.7  7.22 GB    256     7.7%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC205
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns   Host ID                               Rack
UN  10.1.0.1   6.04 GB    256     8.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC10
UN  10.1.0.2   7.55 GB    256     7.4%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC8
UN  10.1.0.3   5.83 GB    256     7.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC9
UN  10.1.0.4    7.34 GB    256     7.9%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC6
UN  10.1.0.5   7.57 GB    256     8.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC7
UN  10.1.0.6   5.31 GB    256     7.3%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC10
UN  10.1.0.7   5.47 GB    256     8.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC9

I did a cql trace on the query and here is the trace, and it does say 
   Write timeout; received 3 of 4 required replies | 17:27:52,831 |  10.1.0.1 |        2002873

at the end. I guess that is where the client gets the error from. But the rows was inserted to Cassandra correctly. And I traced read with local_quorum and it behaves correctly and the reads don't go to DC2. The problem is only with writes on local_quorum.

Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890

 activity                                                                                                                                    | timestamp    | source      | source_elapsed
---------------------------------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------
                                                                                                                          execute_cql3_query | 17:27:50,828 |  10.1.0.1 |              0
 Parsing insert into test (user_id, created, event_data, event_id)values ( 123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); | 17:27:50,828 |  10.1.0.1 |             39
                                                                                                                         Preparing statement | 17:27:50,828 |  10.1.0.1 |            135
                                                                                                           Message received from /10.1.0.1 | 17:27:50,829 |  10.1.0.5 |             25
                                                                                                              Sending message to /10.1.0.5 | 17:27:50,829 |  10.1.0.1 |            421
                                                                                                   Executing single-partition query on users | 17:27:50,829 |  10.1.0.5 |            177
                                                                                                                Acquiring sstable references | 17:27:50,829 |  10.1.0.5 |            191
                                                                                                                 Merging memtable tombstones | 17:27:50,830 |  10.1.0.5 |            208
                                                                                                           Message received from /10.1.0.5 | 17:27:50,830 |  10.1.0.1 |           1461
                                                                                                           Message received from /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |             14
                                                                                                                 Key cache hit for sstable 1 | 17:27:50,830 |  10.1.0.5 |            254
                                                                                                        Processing response from /10.1.0.5 | 17:27:50,830 |  10.1.0.1 |           1514
                                                                                                   Executing single-partition query on users | 17:27:50,830 |  10.1.0.2 |             78
                                                                                                 Seeking to partition beginning in data file | 17:27:50,830 |  10.1.0.5 |            264
                                                                                                              Sending message to /10.1.0.2 | 17:27:50,830 |  10.1.0.1 |           1517
                                                                                                                Acquiring sstable references | 17:27:50,830 |  10.1.0.2 |             85
                                                                   Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 17:27:50,830 |  10.1.0.5 |            453
                                                                                                           Determining replicas for mutation | 17:27:50,830 |  10.1.0.1 |           1746
                                                                                                                 Merging memtable tombstones | 17:27:50,830 |  10.1.0.2 |             97
                                                                                                  Merging data from memtables and 1 sstables | 17:27:50,830 |  10.1.0.5 |            476
                                                                                                           Message received from /10.1.0.2 | 17:27:50,830 |  10.1.0.1 |           2369
                                                                                                                 Key cache hit for sstable 2 | 17:27:50,830 |  10.1.0.2 |            120
                                                                                                          Read 1 live and 0 tombstoned cells | 17:27:50,830 |  10.1.0.5 |            506
                                                                                                 Seeking to partition beginning in data file | 17:27:50,830 |  10.1.0.2 |            123
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,830 |  10.1.0.5 |            570
                                                                   Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 17:27:50,830 |  10.1.0.2 |            288
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,830 |  10.1.0.5 |            617
                                                                                                  Merging data from memtables and 1 sstables | 17:27:50,830 |  10.1.0.2 |            297
                                                                                                          Read 1 live and 0 tombstoned cells | 17:27:50,830 |  10.1.0.2 |            319
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |            362
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |            420
                                                                                                           Message received from /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |             17
                                                                                                              Sending message to /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2435
                                                                                                           Message received from /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |              8
                                                                                                              Acquiring switchLock read lock | 17:27:50,831 |   10.1.0.4 |             61
                                                                                                        Processing response from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2488
                                                                                                              Acquiring switchLock read lock | 17:27:50,831 |  10.1.0.2 |             44
                                                                                                                      Appending to commitlog | 17:27:50,831 |   10.1.0.4 |             78
                                                                                    Not hinting /10.2.0.4 which has been down 364809650ms | 17:27:50,831 |  10.1.0.1 |           2503
                                                                                                                      Appending to commitlog | 17:27:50,831 |  10.1.0.2 |             62
                                                                                                                    Adding to event memtable | 17:27:50,831 |   10.1.0.4 |             96
                                                                                                               Sending message to /10.1.0.4 | 17:27:50,831 |  10.1.0.1 |           2557
                                                                                                                    Adding to event memtable | 17:27:50,831 |  10.1.0.2 |             80
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |            177
                                                                                                              Acquiring switchLock read lock | 17:27:50,831 |  10.1.0.1 |           2669
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |            160
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |            333
                                                                                                                      Appending to commitlog | 17:27:50,831 |  10.1.0.1 |           2682
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |            266
                                                                                                                    Adding to event memtable | 17:27:50,831 |  10.1.0.1 |           2720
                                                                                                             Sending message to /10.2.0.5 | 17:27:50,831 |  10.1.0.1 |           2758
                                                                                                           Message received from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2989
                                                                                                        Processing response from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           3024
                                                                                                            Message received from /10.1.0.4 | 17:27:50,832 |  10.1.0.1 |           3764
                                                                                                         Processing response from /10.1.0.4 | 17:27:50,832 |  10.1.0.1 |           3805
                                                                                                           Message received from /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |             24
                                                                                                   Enqueuing forwarded write to /10.2.0.7 | 17:27:50,841 | 10.2.0.5 |            255
                                                                                                   Enqueuing forwarded write to /10.2.0.3 | 17:27:50,841 | 10.2.0.5 |            283
                                                                                                    Enqueuing forwarded write to /10.2.0.6 | 17:27:50,841 | 10.2.0.5 |            307
                                                                                                              Acquiring switchLock read lock | 17:27:50,841 | 10.2.0.5 |            362
                                                                                                                      Appending to commitlog | 17:27:50,841 | 10.2.0.5 |            380
                                                                                                             Sending message to /10.2.0.7 | 17:27:50,841 | 10.2.0.5 |            382
                                                                                                                    Adding to event memtable | 17:27:50,841 | 10.2.0.5 |            411
                                                                                                              Sending message to /10.2.0.6 | 17:27:50,841 | 10.2.0.5 |            429
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |            484
                                                                                                             Sending message to /10.2.0.3 | 17:27:50,841 | 10.2.0.5 |            561
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |            625
                                                                                                              Acquiring switchLock read lock | 17:27:50,842 |  10.2.0.6 |           1031
                                                                                                              Acquiring switchLock read lock | 17:27:50,842 | 10.2.0.7 |            178
                                                                                                                      Appending to commitlog | 17:27:50,842 |  10.2.0.6 |           1066
                                                                                                                      Appending to commitlog | 17:27:50,842 | 10.2.0.7 |            196
                                                                                                                    Adding to event memtable | 17:27:50,842 |  10.2.0.6 |           1118
                                                                                                                    Adding to event memtable | 17:27:50,842 | 10.2.0.7 |            231
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,842 |  10.2.0.6 |           1181
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,842 | 10.2.0.7 |            286
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,842 | 10.2.0.7 |            421
                                                                                                              Acquiring switchLock read lock | 17:27:50,843 | 10.2.0.3 |           1216
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,843 |  10.2.0.6 |           1382
                                                                                                                      Appending to commitlog | 17:27:50,843 | 10.2.0.3 |           1239
                                                                                                                    Adding to event memtable | 17:27:50,843 | 10.2.0.3 |           1313
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,843 | 10.2.0.3 |           1407
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,843 | 10.2.0.3 |           1631
                                                                                                          Message received from /10.2.0.5 | 17:27:50,851 |  10.1.0.1 |          23333
                                                                                                       Processing response from /10.2.0.5 | 17:27:50,851 |  10.1.0.1 |          23380
                                                                                                          Message received from /10.2.0.7 | 17:27:50,852 |  10.1.0.1 |          23908
                                                                                                       Processing response from /10.2.0.7 | 17:27:50,852 |  10.1.0.1 |          23953
                                                                                                           Message received from /10.2.0.6 | 17:27:50,853 |  10.1.0.1 |          25143
                                                                                                        Processing response from /10.2.0.6 | 17:27:50,853 |  10.1.0.1 |          25178
                                                                                                          Message received from /10.2.0.3 | 17:27:50,854 |  10.1.0.1 |          25478
                                                                                                       Processing response from /10.2.0.3 | 17:27:50,854 |  10.1.0.1 |          25516
                                                                                             Write timeout; received 3 of 4 required replies | 17:27:52,831 |  10.1.0.1 |        2002873
                                                                                                                            Request complete | 17:27:52,833 |  10.1.0.1 |        2005989

cqlsh:XXXX> CONSISTENCY
Current consistency level is LOCAL_QUORUM.

  was:
We have two DC3, each with 7 nodes.
Here is the keyspace setup:

 create keyspace test
 with placement_strategy = 'NetworkTopologyStrategy'
 and strategy_options = {DC2 : 3, DC1 : 3}
 and durable_writes = true;

We brought down two nodes in DC2 for maintenance. We only write to DC1 using local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to client since local_quorum should succeed.

Here are the output from nodetool status

Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns   Host ID                               Rack
UN  10.2.0.1  10.92 GB   256     7.9%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC206
UN  10.2.0.2   6.17 GB    256     8.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC106
UN  10.2.0.3  6.63 GB    256     7.3%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC107
DL  10.2.0.4  1.54 GB    256     7.7%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC107
UN  10.2.0.5  6.02 GB    256     6.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC106
UJ  10.2.0.6   3.68 GB    256     ?      XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC205
UN  10.2.0.7  7.22 GB    256     7.7%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC205
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns   Host ID                               Rack
UN  10.1.0.1   6.04 GB    256     8.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC10
UN  10.1.0.2   7.55 GB    256     7.4%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC8
UN  10.1.0.3   5.83 GB    256     7.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC9
UN  10.1.0.4    7.34 GB    256     7.9%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC6
UN  10.1.0.5   7.57 GB    256     8.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC7
UN  10.1.0.6   5.31 GB    256     7.3%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC10
UN  10.1.0.7   5.47 GB    256     8.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC9

I did a cql trace on the query and here is the trace, and it does say 
   Write timeout; received 3 of 4 required replies | 17:27:52,831 |  10.1.0.1 |        2002873

at the end. I guess that is where the client gets the error from.

Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890

 activity                                                                                                                                    | timestamp    | source      | source_elapsed
---------------------------------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------
                                                                                                                          execute_cql3_query | 17:27:50,828 |  10.1.0.1 |              0
 Parsing insert into test (user_id, created, event_data, event_id)values ( 123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); | 17:27:50,828 |  10.1.0.1 |             39
                                                                                                                         Preparing statement | 17:27:50,828 |  10.1.0.1 |            135
                                                                                                           Message received from /10.1.0.1 | 17:27:50,829 |  10.1.0.5 |             25
                                                                                                              Sending message to /10.1.0.5 | 17:27:50,829 |  10.1.0.1 |            421
                                                                                                   Executing single-partition query on users | 17:27:50,829 |  10.1.0.5 |            177
                                                                                                                Acquiring sstable references | 17:27:50,829 |  10.1.0.5 |            191
                                                                                                                 Merging memtable tombstones | 17:27:50,830 |  10.1.0.5 |            208
                                                                                                           Message received from /10.1.0.5 | 17:27:50,830 |  10.1.0.1 |           1461
                                                                                                           Message received from /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |             14
                                                                                                                 Key cache hit for sstable 1 | 17:27:50,830 |  10.1.0.5 |            254
                                                                                                        Processing response from /10.1.0.5 | 17:27:50,830 |  10.1.0.1 |           1514
                                                                                                   Executing single-partition query on users | 17:27:50,830 |  10.1.0.2 |             78
                                                                                                 Seeking to partition beginning in data file | 17:27:50,830 |  10.1.0.5 |            264
                                                                                                              Sending message to /10.1.0.2 | 17:27:50,830 |  10.1.0.1 |           1517
                                                                                                                Acquiring sstable references | 17:27:50,830 |  10.1.0.2 |             85
                                                                   Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 17:27:50,830 |  10.1.0.5 |            453
                                                                                                           Determining replicas for mutation | 17:27:50,830 |  10.1.0.1 |           1746
                                                                                                                 Merging memtable tombstones | 17:27:50,830 |  10.1.0.2 |             97
                                                                                                  Merging data from memtables and 1 sstables | 17:27:50,830 |  10.1.0.5 |            476
                                                                                                           Message received from /10.1.0.2 | 17:27:50,830 |  10.1.0.1 |           2369
                                                                                                                 Key cache hit for sstable 2 | 17:27:50,830 |  10.1.0.2 |            120
                                                                                                          Read 1 live and 0 tombstoned cells | 17:27:50,830 |  10.1.0.5 |            506
                                                                                                 Seeking to partition beginning in data file | 17:27:50,830 |  10.1.0.2 |            123
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,830 |  10.1.0.5 |            570
                                                                   Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 17:27:50,830 |  10.1.0.2 |            288
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,830 |  10.1.0.5 |            617
                                                                                                  Merging data from memtables and 1 sstables | 17:27:50,830 |  10.1.0.2 |            297
                                                                                                          Read 1 live and 0 tombstoned cells | 17:27:50,830 |  10.1.0.2 |            319
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |            362
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |            420
                                                                                                           Message received from /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |             17
                                                                                                              Sending message to /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2435
                                                                                                           Message received from /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |              8
                                                                                                              Acquiring switchLock read lock | 17:27:50,831 |   10.1.0.4 |             61
                                                                                                        Processing response from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2488
                                                                                                              Acquiring switchLock read lock | 17:27:50,831 |  10.1.0.2 |             44
                                                                                                                      Appending to commitlog | 17:27:50,831 |   10.1.0.4 |             78
                                                                                    Not hinting /10.2.0.4 which has been down 364809650ms | 17:27:50,831 |  10.1.0.1 |           2503
                                                                                                                      Appending to commitlog | 17:27:50,831 |  10.1.0.2 |             62
                                                                                                                    Adding to event memtable | 17:27:50,831 |   10.1.0.4 |             96
                                                                                                               Sending message to /10.1.0.4 | 17:27:50,831 |  10.1.0.1 |           2557
                                                                                                                    Adding to event memtable | 17:27:50,831 |  10.1.0.2 |             80
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |            177
                                                                                                              Acquiring switchLock read lock | 17:27:50,831 |  10.1.0.1 |           2669
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |            160
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |            333
                                                                                                                      Appending to commitlog | 17:27:50,831 |  10.1.0.1 |           2682
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |            266
                                                                                                                    Adding to event memtable | 17:27:50,831 |  10.1.0.1 |           2720
                                                                                                             Sending message to /10.2.0.5 | 17:27:50,831 |  10.1.0.1 |           2758
                                                                                                           Message received from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2989
                                                                                                        Processing response from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           3024
                                                                                                            Message received from /10.1.0.4 | 17:27:50,832 |  10.1.0.1 |           3764
                                                                                                         Processing response from /10.1.0.4 | 17:27:50,832 |  10.1.0.1 |           3805
                                                                                                           Message received from /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |             24
                                                                                                   Enqueuing forwarded write to /10.2.0.7 | 17:27:50,841 | 10.2.0.5 |            255
                                                                                                   Enqueuing forwarded write to /10.2.0.3 | 17:27:50,841 | 10.2.0.5 |            283
                                                                                                    Enqueuing forwarded write to /10.2.0.6 | 17:27:50,841 | 10.2.0.5 |            307
                                                                                                              Acquiring switchLock read lock | 17:27:50,841 | 10.2.0.5 |            362
                                                                                                                      Appending to commitlog | 17:27:50,841 | 10.2.0.5 |            380
                                                                                                             Sending message to /10.2.0.7 | 17:27:50,841 | 10.2.0.5 |            382
                                                                                                                    Adding to event memtable | 17:27:50,841 | 10.2.0.5 |            411
                                                                                                              Sending message to /10.2.0.6 | 17:27:50,841 | 10.2.0.5 |            429
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |            484
                                                                                                             Sending message to /10.2.0.3 | 17:27:50,841 | 10.2.0.5 |            561
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |            625
                                                                                                              Acquiring switchLock read lock | 17:27:50,842 |  10.2.0.6 |           1031
                                                                                                              Acquiring switchLock read lock | 17:27:50,842 | 10.2.0.7 |            178
                                                                                                                      Appending to commitlog | 17:27:50,842 |  10.2.0.6 |           1066
                                                                                                                      Appending to commitlog | 17:27:50,842 | 10.2.0.7 |            196
                                                                                                                    Adding to event memtable | 17:27:50,842 |  10.2.0.6 |           1118
                                                                                                                    Adding to event memtable | 17:27:50,842 | 10.2.0.7 |            231
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,842 |  10.2.0.6 |           1181
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,842 | 10.2.0.7 |            286
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,842 | 10.2.0.7 |            421
                                                                                                              Acquiring switchLock read lock | 17:27:50,843 | 10.2.0.3 |           1216
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,843 |  10.2.0.6 |           1382
                                                                                                                      Appending to commitlog | 17:27:50,843 | 10.2.0.3 |           1239
                                                                                                                    Adding to event memtable | 17:27:50,843 | 10.2.0.3 |           1313
                                                                                                           Enqueuing response to /10.1.0.1 | 17:27:50,843 | 10.2.0.3 |           1407
                                                                                                              Sending message to /10.1.0.1 | 17:27:50,843 | 10.2.0.3 |           1631
                                                                                                          Message received from /10.2.0.5 | 17:27:50,851 |  10.1.0.1 |          23333
                                                                                                       Processing response from /10.2.0.5 | 17:27:50,851 |  10.1.0.1 |          23380
                                                                                                          Message received from /10.2.0.7 | 17:27:50,852 |  10.1.0.1 |          23908
                                                                                                       Processing response from /10.2.0.7 | 17:27:50,852 |  10.1.0.1 |          23953
                                                                                                           Message received from /10.2.0.6 | 17:27:50,853 |  10.1.0.1 |          25143
                                                                                                        Processing response from /10.2.0.6 | 17:27:50,853 |  10.1.0.1 |          25178
                                                                                                          Message received from /10.2.0.3 | 17:27:50,854 |  10.1.0.1 |          25478
                                                                                                       Processing response from /10.2.0.3 | 17:27:50,854 |  10.1.0.1 |          25516
                                                                                             Write timeout; received 3 of 4 required replies | 17:27:52,831 |  10.1.0.1 |        2002873
                                                                                                                            Request complete | 17:27:52,833 |  10.1.0.1 |        2005989




> LOCAL_QUORUM writes returns wrong message
> -----------------------------------------
>
>                 Key: CASSANDRA-8819
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8819
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 6.6
>            Reporter: Wei Zhu
>             Fix For: 2.0.8
>
>
> We have two DC3, each with 7 nodes.
> Here is the keyspace setup:
>  create keyspace test
>  with placement_strategy = 'NetworkTopologyStrategy'
>  and strategy_options = {DC2 : 3, DC1 : 3}
>  and durable_writes = true;
> We brought down two nodes in DC2 for maintenance. We only write to DC1 using local_quroum (using datastax JavaClient)
> But we see this errors in the log:
> Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica were required but only 3 acknowledged the write
> why does it say 4 replica were required? and Why would it give error back to client since local_quorum should succeed.
> Here are the output from nodetool status
> Note: Ownership information does not include topology; for complete information, specify a keyspace
> Datacenter: DC2
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address      Load       Tokens  Owns   Host ID                               Rack
> UN  10.2.0.1  10.92 GB   256     7.9%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC206
> UN  10.2.0.2   6.17 GB    256     8.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC106
> UN  10.2.0.3  6.63 GB    256     7.3%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC107
> DL  10.2.0.4  1.54 GB    256     7.7%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC107
> UN  10.2.0.5  6.02 GB    256     6.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC106
> UJ  10.2.0.6   3.68 GB    256     ?      XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC205
> UN  10.2.0.7  7.22 GB    256     7.7%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC205
> Datacenter: DC1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address      Load       Tokens  Owns   Host ID                               Rack
> UN  10.1.0.1   6.04 GB    256     8.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC10
> UN  10.1.0.2   7.55 GB    256     7.4%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC8
> UN  10.1.0.3   5.83 GB    256     7.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC9
> UN  10.1.0.4    7.34 GB    256     7.9%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC6
> UN  10.1.0.5   7.57 GB    256     8.0%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC7
> UN  10.1.0.6   5.31 GB    256     7.3%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX  RAC10
> UN  10.1.0.7   5.47 GB    256     8.6%   XXXXXXXXXXXXXXXXXXXXXXXXXXXX RAC9
> I did a cql trace on the query and here is the trace, and it does say 
>    Write timeout; received 3 of 4 required replies | 17:27:52,831 |  10.1.0.1 |        2002873
> at the end. I guess that is where the client gets the error from. But the rows was inserted to Cassandra correctly. And I traced read with local_quorum and it behaves correctly and the reads don't go to DC2. The problem is only with writes on local_quorum.
> Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
>  activity                                                                                                                                    | timestamp    | source      | source_elapsed
> ---------------------------------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------
>                                                                                                                           execute_cql3_query | 17:27:50,828 |  10.1.0.1 |              0
>  Parsing insert into test (user_id, created, event_data, event_id)values ( 123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); | 17:27:50,828 |  10.1.0.1 |             39
>                                                                                                                          Preparing statement | 17:27:50,828 |  10.1.0.1 |            135
>                                                                                                            Message received from /10.1.0.1 | 17:27:50,829 |  10.1.0.5 |             25
>                                                                                                               Sending message to /10.1.0.5 | 17:27:50,829 |  10.1.0.1 |            421
>                                                                                                    Executing single-partition query on users | 17:27:50,829 |  10.1.0.5 |            177
>                                                                                                                 Acquiring sstable references | 17:27:50,829 |  10.1.0.5 |            191
>                                                                                                                  Merging memtable tombstones | 17:27:50,830 |  10.1.0.5 |            208
>                                                                                                            Message received from /10.1.0.5 | 17:27:50,830 |  10.1.0.1 |           1461
>                                                                                                            Message received from /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |             14
>                                                                                                                  Key cache hit for sstable 1 | 17:27:50,830 |  10.1.0.5 |            254
>                                                                                                         Processing response from /10.1.0.5 | 17:27:50,830 |  10.1.0.1 |           1514
>                                                                                                    Executing single-partition query on users | 17:27:50,830 |  10.1.0.2 |             78
>                                                                                                  Seeking to partition beginning in data file | 17:27:50,830 |  10.1.0.5 |            264
>                                                                                                               Sending message to /10.1.0.2 | 17:27:50,830 |  10.1.0.1 |           1517
>                                                                                                                 Acquiring sstable references | 17:27:50,830 |  10.1.0.2 |             85
>                                                                    Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 17:27:50,830 |  10.1.0.5 |            453
>                                                                                                            Determining replicas for mutation | 17:27:50,830 |  10.1.0.1 |           1746
>                                                                                                                  Merging memtable tombstones | 17:27:50,830 |  10.1.0.2 |             97
>                                                                                                   Merging data from memtables and 1 sstables | 17:27:50,830 |  10.1.0.5 |            476
>                                                                                                            Message received from /10.1.0.2 | 17:27:50,830 |  10.1.0.1 |           2369
>                                                                                                                  Key cache hit for sstable 2 | 17:27:50,830 |  10.1.0.2 |            120
>                                                                                                           Read 1 live and 0 tombstoned cells | 17:27:50,830 |  10.1.0.5 |            506
>                                                                                                  Seeking to partition beginning in data file | 17:27:50,830 |  10.1.0.2 |            123
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,830 |  10.1.0.5 |            570
>                                                                    Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | 17:27:50,830 |  10.1.0.2 |            288
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,830 |  10.1.0.5 |            617
>                                                                                                   Merging data from memtables and 1 sstables | 17:27:50,830 |  10.1.0.2 |            297
>                                                                                                           Read 1 live and 0 tombstoned cells | 17:27:50,830 |  10.1.0.2 |            319
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |            362
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,830 |  10.1.0.2 |            420
>                                                                                                            Message received from /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |             17
>                                                                                                               Sending message to /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2435
>                                                                                                            Message received from /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |              8
>                                                                                                               Acquiring switchLock read lock | 17:27:50,831 |   10.1.0.4 |             61
>                                                                                                         Processing response from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2488
>                                                                                                               Acquiring switchLock read lock | 17:27:50,831 |  10.1.0.2 |             44
>                                                                                                                       Appending to commitlog | 17:27:50,831 |   10.1.0.4 |             78
>                                                                                     Not hinting /10.2.0.4 which has been down 364809650ms | 17:27:50,831 |  10.1.0.1 |           2503
>                                                                                                                       Appending to commitlog | 17:27:50,831 |  10.1.0.2 |             62
>                                                                                                                     Adding to event memtable | 17:27:50,831 |   10.1.0.4 |             96
>                                                                                                                Sending message to /10.1.0.4 | 17:27:50,831 |  10.1.0.1 |           2557
>                                                                                                                     Adding to event memtable | 17:27:50,831 |  10.1.0.2 |             80
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |            177
>                                                                                                               Acquiring switchLock read lock | 17:27:50,831 |  10.1.0.1 |           2669
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |            160
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,831 |   10.1.0.4 |            333
>                                                                                                                       Appending to commitlog | 17:27:50,831 |  10.1.0.1 |           2682
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,831 |  10.1.0.2 |            266
>                                                                                                                     Adding to event memtable | 17:27:50,831 |  10.1.0.1 |           2720
>                                                                                                              Sending message to /10.2.0.5 | 17:27:50,831 |  10.1.0.1 |           2758
>                                                                                                            Message received from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           2989
>                                                                                                         Processing response from /10.1.0.2 | 17:27:50,831 |  10.1.0.1 |           3024
>                                                                                                             Message received from /10.1.0.4 | 17:27:50,832 |  10.1.0.1 |           3764
>                                                                                                          Processing response from /10.1.0.4 | 17:27:50,832 |  10.1.0.1 |           3805
>                                                                                                            Message received from /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |             24
>                                                                                                    Enqueuing forwarded write to /10.2.0.7 | 17:27:50,841 | 10.2.0.5 |            255
>                                                                                                    Enqueuing forwarded write to /10.2.0.3 | 17:27:50,841 | 10.2.0.5 |            283
>                                                                                                     Enqueuing forwarded write to /10.2.0.6 | 17:27:50,841 | 10.2.0.5 |            307
>                                                                                                               Acquiring switchLock read lock | 17:27:50,841 | 10.2.0.5 |            362
>                                                                                                                       Appending to commitlog | 17:27:50,841 | 10.2.0.5 |            380
>                                                                                                              Sending message to /10.2.0.7 | 17:27:50,841 | 10.2.0.5 |            382
>                                                                                                                     Adding to event memtable | 17:27:50,841 | 10.2.0.5 |            411
>                                                                                                               Sending message to /10.2.0.6 | 17:27:50,841 | 10.2.0.5 |            429
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |            484
>                                                                                                              Sending message to /10.2.0.3 | 17:27:50,841 | 10.2.0.5 |            561
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,841 | 10.2.0.5 |            625
>                                                                                                               Acquiring switchLock read lock | 17:27:50,842 |  10.2.0.6 |           1031
>                                                                                                               Acquiring switchLock read lock | 17:27:50,842 | 10.2.0.7 |            178
>                                                                                                                       Appending to commitlog | 17:27:50,842 |  10.2.0.6 |           1066
>                                                                                                                       Appending to commitlog | 17:27:50,842 | 10.2.0.7 |            196
>                                                                                                                     Adding to event memtable | 17:27:50,842 |  10.2.0.6 |           1118
>                                                                                                                     Adding to event memtable | 17:27:50,842 | 10.2.0.7 |            231
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,842 |  10.2.0.6 |           1181
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,842 | 10.2.0.7 |            286
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,842 | 10.2.0.7 |            421
>                                                                                                               Acquiring switchLock read lock | 17:27:50,843 | 10.2.0.3 |           1216
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,843 |  10.2.0.6 |           1382
>                                                                                                                       Appending to commitlog | 17:27:50,843 | 10.2.0.3 |           1239
>                                                                                                                     Adding to event memtable | 17:27:50,843 | 10.2.0.3 |           1313
>                                                                                                            Enqueuing response to /10.1.0.1 | 17:27:50,843 | 10.2.0.3 |           1407
>                                                                                                               Sending message to /10.1.0.1 | 17:27:50,843 | 10.2.0.3 |           1631
>                                                                                                           Message received from /10.2.0.5 | 17:27:50,851 |  10.1.0.1 |          23333
>                                                                                                        Processing response from /10.2.0.5 | 17:27:50,851 |  10.1.0.1 |          23380
>                                                                                                           Message received from /10.2.0.7 | 17:27:50,852 |  10.1.0.1 |          23908
>                                                                                                        Processing response from /10.2.0.7 | 17:27:50,852 |  10.1.0.1 |          23953
>                                                                                                            Message received from /10.2.0.6 | 17:27:50,853 |  10.1.0.1 |          25143
>                                                                                                         Processing response from /10.2.0.6 | 17:27:50,853 |  10.1.0.1 |          25178
>                                                                                                           Message received from /10.2.0.3 | 17:27:50,854 |  10.1.0.1 |          25478
>                                                                                                        Processing response from /10.2.0.3 | 17:27:50,854 |  10.1.0.1 |          25516
>                                                                                              Write timeout; received 3 of 4 required replies | 17:27:52,831 |  10.1.0.1 |        2002873
>                                                                                                                             Request complete | 17:27:52,833 |  10.1.0.1 |        2005989
> cqlsh:XXXX> CONSISTENCY
> Current consistency level is LOCAL_QUORUM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message