cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thibaut (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
Date Wed, 02 Feb 2011 16:09:29 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989689#comment-12989689
] 

Thibaut commented on CASSANDRA-2081:
------------------------------------

Debug Output:



========================================
192.168.0.1
--------------------
2011-02-02 12:54:29,117 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating
load info ...

2011-02-02 12:54:35,016 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=}
to 567@/192.168.0.7
2011-02-02 12:54:45,015 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=}
to 600@/192.168.0.7

2011-02-02 12:54:49,679 DEBUG [pool-1-thread-5] ClientState.java (line 91) logged out: #<User
allow_all groups=[]>

========================================
192.168.0.2
--------------------
2011-02-02 12:54:30,147 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating
load info ...

2011-02-02 12:54:34,789 DEBUG [MutationStage:5] RowMutationVerbHandler.java (line 52) Applying
RowMutation(keyspace='table_lists', key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
2011-02-02 12:54:34,790 DEBUG [MutationStage:5] Table.java (line 397) applying mutation of
row 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
2011-02-02 12:54:34,792 DEBUG [MutationStage:5] RowMutationVerbHandler.java (line 81) RowMutation(keyspace='table_lists',
key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
applied.  Sending response to 566@/192.168.0.7

2011-02-02 12:54:34,953 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=}
to 567@/192.168.0.7
2011-02-02 12:54:44,965 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=}
to 600@/192.168.0.7

2011-02-02 12:54:45,057 DEBUG [ScheduledTasks:1] GCInspector.java (line 135) GC for ParNew:
14 ms, 13086296 reclaimed leaving 2087311488 used; max is 4856348672
2011-02-02 12:54:49,613 DEBUG [pool-1-thread-11] ClientState.java (line 91) logged out: #<User
allow_all groups=[]>

========================================
192.168.0.3
--------------------
2011-02-02 12:54:30,920 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating
load info ...

2011-02-02 12:54:34,368 DEBUG [MutationStage:8] RowMutationVerbHandler.java (line 52) Applying
RowMutation(keyspace='table_lists', key='36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165',
modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674314000!2419199,]),
ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674322000!2419199,])])
2011-02-02 12:54:34,369 DEBUG [MutationStage:8] Table.java (line 397) applying mutation of
row 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
2011-02-02 12:54:34,371 DEBUG [MutationStage:8] RowMutationVerbHandler.java (line 81) RowMutation(keyspace='table_lists',
key='36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165',
modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674314000!2419199,]),
ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674322000!2419199,])])
applied.  Sending response to 562@/192.168.0.7
2011-02-02 12:54:34,381 DEBUG [MutationStage:9] RowMutationVerbHandler.java (line 52) Applying
RowMutation(keyspace='table_lists', key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
2011-02-02 12:54:34,382 DEBUG [MutationStage:9] Table.java (line 397) applying mutation of
row 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
2011-02-02 12:54:34,383 DEBUG [MutationStage:9] RowMutationVerbHandler.java (line 81) RowMutation(keyspace='table_lists',
key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
applied.  Sending response to 564@/192.168.0.7

2011-02-02 12:54:34,539 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=}
to 567@/192.168.0.7
2011-02-02 12:54:44,540 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=}
to 600@/192.168.0.7

2011-02-02 12:54:49,202 DEBUG [pool-1-thread-13] ClientState.java (line 91) logged out: #<User
allow_all groups=[]>

========================================
192.168.0.7
--------------------
2011-02-02 12:54:34,059 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating
load info ...

2011-02-02 12:54:34,356 DEBUG [pool-1-thread-1] CassandraServer.java (line 355) batch_mutate
2011-02-02 12:54:34,375 DEBUG [pool-1-thread-1] StorageProxy.java (line 154) insert writing
key 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
to 561@/192.168.0.5
2011-02-02 12:54:34,375 DEBUG [pool-1-thread-1] StorageProxy.java (line 154) insert writing
key 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
to 562@/192.168.0.3
2011-02-02 12:54:34,381 DEBUG [pool-1-thread-1] StorageProxy.java (line 154) insert writing
key 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
to 563@/94.242.198.13
2011-02-02 12:54:34,384 DEBUG [RequestResponseStage:3] ResponseVerbHandler.java (line 48)
Processing response on a callback from 561@/192.168.0.5
2011-02-02 12:54:34,386 DEBUG [RequestResponseStage:5] ResponseVerbHandler.java (line 48)
Processing response on a callback from 563@/192.168.0.4
2011-02-02 12:54:34,386 DEBUG [RequestResponseStage:4] ResponseVerbHandler.java (line 48)
Processing response on a callback from 562@/192.168.0.3

2011-02-02 12:54:34,391 DEBUG [pool-1-thread-2] CassandraServer.java (line 355) batch_mutate
2011-02-02 12:54:34,393 DEBUG [pool-1-thread-2] StorageProxy.java (line 154) insert writing
key 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
to 564@/192.168.0.3
2011-02-02 12:54:34,394 DEBUG [pool-1-thread-2] StorageProxy.java (line 154) insert writing
key 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
to 565@/192.168.0.4
2011-02-02 12:54:34,394 DEBUG [pool-1-thread-2] StorageProxy.java (line 154) insert writing
key 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
to 566@/192.168.0.2
2011-02-02 12:54:34,398 DEBUG [RequestResponseStage:6] ResponseVerbHandler.java (line 48)
Processing response on a callback from 564@/192.168.0.3
2011-02-02 12:54:34,398 DEBUG [RequestResponseStage:7] ResponseVerbHandler.java (line 48)
Processing response on a callback from 565@/192.168.0.4
2011-02-02 12:54:34,399 DEBUG [RequestResponseStage:8] ResponseVerbHandler.java (line 48)
Processing response on a callback from 566@/192.168.0.2

2011-02-02 12:54:34,531 DEBUG [pool-1-thread-3] CassandraServer.java (line 445) range_slice
2011-02-02 12:54:34,533 DEBUG [pool-1-thread-3] StorageProxy.java (line 514) RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,], max_keys=250}
2011-02-02 12:54:34,534 DEBUG [pool-1-thread-3] StorageProxy.java (line 705) restricted ranges
for query [,] are [[,24], (24,49], (49,6d], (6d,92], (92,b6], (b6,db], (db,ffffffffffffffff],
(ffffffffffffffff,]]
2011-02-02 12:54:34,536 DEBUG [pool-1-thread-3] ReadCallback.java (line 58) ReadCallback blocking
for 2 responses
2011-02-02 12:54:34,536 DEBUG [pool-1-thread-3] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.1
2011-02-02 12:54:34,537 DEBUG [pool-1-thread-3] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.2
2011-02-02 12:54:34,537 DEBUG [pool-1-thread-3] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.3
2011-02-02 12:54:34,554 DEBUG [RequestResponseStage:1] ResponseVerbHandler.java (line 48)
Processing response on a callback from 567@/192.168.0.3
2011-02-02 12:54:44,537 DEBUG [pool-1-thread-3] CassandraServer.java (line 483) ... timed
out
2011-02-02 12:54:44,548 DEBUG [pool-1-thread-3] ClientState.java (line 91) logged out: #<User
allow_all groups=[]>

2011-02-02 12:54:44,549 DEBUG [pool-1-thread-4] CassandraServer.java (line 445) range_slice
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] StorageProxy.java (line 514) RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,], max_keys=250}
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] StorageProxy.java (line 705) restricted ranges
for query [,] are [[,24], (24,49], (49,6d], (6d,92], (92,b6], (b6,db], (db,ffffffffffffffff],
(ffffffffffffffff,]]
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] ReadCallback.java (line 58) ReadCallback blocking
for 2 responses
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.1
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.2
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.3
2011-02-02 12:54:44,554 DEBUG [RequestResponseStage:5] ResponseVerbHandler.java (line 48)
Processing response on a callback from 600@/192.168.0.1
2011-02-02 12:54:54,550 DEBUG [pool-1-thread-4] CassandraServer.java (line 483) ... timed
out
2011-02-02 12:54:54,552 DEBUG [pool-1-thread-4] ClientState.java (line 91) logged out: #<User
allow_all groups=[]>




> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2081
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: linux, hector + cassandra
>            Reporter: Thibaut
>            Priority: Blocker
>             Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level
ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar),
I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar
and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160
host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException:
No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status
false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on
this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra
cli.
> 192.168.0.1     Up     Normal  8.02 GB         5.00%   0cc
> 192.168.0.2     Up     Normal  7.96 GB         5.00%   199
> 192.168.0.3     Up     Normal  8.24 GB         5.00%   266
> 192.168.0.4     Up     Normal  4.94 GB         5.00%   333
> 192.168.0.5     Up     Normal  5.02 GB         5.00%   400
> 192.168.0.6     Up     Normal  5 GB            5.00%   4cc
> 192.168.0.7     Up     Normal  5.1 GB          5.00%   599
> 192.168.0.8     Up     Normal  5.07 GB         5.00%   666
> 192.168.0.9     Up     Normal  4.78 GB         5.00%   733
> 192.168.0.10    Up     Normal  4.34 GB         5.00%   7ff
> 192.168.0.11    Up     Normal  5.01 GB         5.00%   8cc
> 192.168.0.12    Up     Normal  5.31 GB         5.00%   999
> 192.168.0.13    Up     Normal  5.56 GB         5.00%   a66
> 192.168.0.14    Up     Normal  5.82 GB         5.00%   b33
> 192.168.0.15    Up     Normal  5.57 GB         5.00%   c00
> 192.168.0.16    Up     Normal  5.03 GB         5.00%   ccc
> 192.168.0.17    Up     Normal  4.77 GB         5.00%   d99
> 192.168.0.18    Down   Normal  ?               5.00%   e66
> 192.168.0.19    Up     Normal  4.78 GB         5.00%   f33
> 192.168.0.20    Up     Normal  4.83 GB         5.00%   ffffffffffffffff

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message