cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thibaut (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
Date Tue, 01 Feb 2011 11:06:29 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989150#comment-12989150
] 

Thibaut commented on CASSANDRA-2081:
------------------------------------

Brandon, I haven't yet run stress test. I can reproduce this error every single time with
a single thread accessing my idle cluster.

I also reverted to an older version of hector, but this won't help. As noted before, this
error doesn't occur running apache-cassandra-2011-01-24_06-01-26.jar.

Here is the debug output of one of the nodes timing out in my application and not returning
an answer:

I only try to access (read/iterator) one single table  "table_usersources".

The application runs on node intr1n5 (192.168.0.5) and I added the debug output of intr1n19
(192.168.0.19). The node intr1n18(192.168.0.18) is down and not responding.

Please let me know if you need more information in order to fix this bug. Thanks!


Intr1n19:

 INFO [HintedHandoff:1] 2011-02-01 11:47:39,772 HintedHandOffManager.java (line 249) Finished
hinted handoff of 0 rows to endpoint /192.168.0.15
 INFO [ScheduledTasks:1] 2011-02-01 11:47:40,773 Gossiper.java (line 205) InetAddress /192.168.0.10
is now dead.
DEBUG [ScheduledTasks:1] 2011-02-01 11:47:40,773 MessagingService.java (line 176) Resetting
pool for /192.168.0.10
 INFO [HintedHandoff:1] 2011-02-01 11:47:40,775 HintedHandOffManager.java (line 192) Started
hinted handoff for endpoint /192.168.0.10
 INFO [GossipStage:1] 2011-02-01 11:47:40,775 Gossiper.java (line 579) InetAddress /192.168.0.10
is now UP
 INFO [HintedHandoff:1] 2011-02-01 11:47:40,775 HintedHandOffManager.java (line 249) Finished
hinted handoff of 0 rows to endpoint /192.168.0.10
DEBUG [WRITE-intr1n4/192.168.0.4] 2011-02-01 11:47:41,479 OutboundTcpConnection.java (line
159) attempting to connect to intr1n4/192.168.0.4
DEBUG [WRITE-intr1n20/192.168.0.20] 2011-02-01 11:47:42,729 OutboundTcpConnection.java (line
159) attempting to connect to intr1n20/192.168.0.20
DEBUG [WRITE-intr1n11/192.168.0.11] 2011-02-01 11:47:42,776 OutboundTcpConnection.java (line
159) attempting to connect to intr1n11/192.168.0.11
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:47:46,781 OutboundTcpConnection.java (line
159) attempting to connect to intr1n18/192.168.0.18
DEBUG [WRITE-intr1n15/192.168.0.15] 2011-02-01 11:47:50,786 OutboundTcpConnection.java (line
159) attempting to connect to intr1n15/192.168.0.15
DEBUG [WRITE-intr1n10/192.168.0.10] 2011-02-01 11:47:53,698 OutboundTcpConnection.java (line
159) attempting to connect to intr1n10/192.168.0.10

DEBUG [ScheduledTasks:1] 2011-02-01 11:48:15,413 GCInspector.java (line 135) GC for ParNew:
32 ms, 124741344 reclaimed leaving 880060312 used; max is 3289776128
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,853 FileUtils.java (line 48) Deleting LocationInfo-f-79-Index.db
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,853 FileUtils.java (line 48) Deleting LocationInfo-f-79-Filter.db
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,854 FileUtils.java (line 48) Deleting LocationInfo-f-79-Statistics.db
 INFO [ScheduledTasks:1] 2011-02-01 11:48:24,854 SSTable.java (line 147) Deleted /hd2/cassandra_md5/data/system/LocationInfo-f-79
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:28,820 OutboundTcpConnection.java (line
159) attempting to connect to intr1n18/192.168.0.18
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,936 CassandraServer.java (line 445) range_slice
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,943 StorageProxy.java (line 514) RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,], max_keys=250}
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,943 StorageProxy.java (line 705) restricted ranges
for query [,] are [[,0cc], (0cc,199], (199,266], (266,333], (333,400], (400,4cc], (4cc,599],
(599,666], (666,733], (733,7ff], (7ff,8cc], (8cc,999], (999,a66], (a66,b33], (b33,c00], (c00,ccc],
(ccc,d99], (d99,e66], (e66,f33], (f33,ffffffffffffffff], (ffffffffffffffff,]]
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 ReadCallback.java (line 58) ReadCallback blocking
for 2 responses
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.1
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.2
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,950 StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76
lim=88 cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.3
DEBUG [RequestResponseStage:1] 2011-02-01 11:48:28,954 ResponseVerbHandler.java (line 48)
Processing response on a callback from 269@/192.168.0.1
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:38,771 StorageLoadBalancer.java (line 349) Disseminating
load info ...
DEBUG [pool-1-thread-1] 2011-02-01 11:48:38,950 CassandraServer.java (line 483) ... timed
out
DEBUG [pool-1-thread-1] 2011-02-01 11:48:38,957 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:44,835 OutboundTcpConnection.java (line
159) attempting to connect to intr1n18/192.168.0.18
DEBUG [pool-1-thread-3] 2011-02-01 11:48:56,275 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-6] 2011-02-01 11:48:56,275 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-7] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-11] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-5] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-2] 2011-02-01 11:48:56,277 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-8] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-10] 2011-02-01 11:48:56,277 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-12] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-15] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-14] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-9] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-16] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-13] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [pool-1-thread-4] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User
allow_all groups=[]>
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:57,845 OutboundTcpConnection.java (line
159) attempting to connect to intr1n18/192.168.0.18


Application:

11:48:25,616 INFO  ~ Registering JMX me.prettyprint.cassandra.service:ServiceType=hector,MonitorType=hector
11:48:25,652 INFO  ~ get connection for table_lists: consistency: ONE
11:48:25,695 INFO  ~ get connection for table_lists: consistency: ONE
11:48:25,825 INFO  ~ Downed Host Retry service started with queue size -1 and retry delay
10s
11:48:28,887 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route
to host
        at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
        at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
        at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
        at me.prettyprint.cassandra.connection.ConcurrentHClientPool.<init>(ConcurrentHClientPool.java:44)
        at me.prettyprint.cassandra.connection.HConnectionManager.<init>(HConnectionManager.java:63)
        at me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:62)
        at me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:58)
        at me.prettyprint.cassandra.service.ThriftCluster.<init>(ThriftCluster.java:17)
        at me.prettyprint.hector.api.factory.HFactory.createCluster(HFactory.java:157)
        at me.prettyprint.hector.api.factory.HFactory.getOrCreateCluster(HFactory.java:136)
     ....
Caused by: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
        ... 23 more
11:48:28,889 ERROR ~ Could not start connection pool for host intr1n18(192.168.0.18):9160
11:48:28,889 INFO  ~ Host detected as down was added to retry queue: intr1n18(192.168.0.18):9160
11:48:28,897 INFO  ~ get connection for table_usersources: consistency: QUORUM
11:48:38,014 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route
to host
        at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
        at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
        at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
        at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.verifyConnection(CassandraHostRetryService.java:116)
        at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.run(CassandraHostRetryService.java:96)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
        ... 13 more
11:48:38,019 ERROR ~ Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable
to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route
to host
11:48:38,020 INFO  ~ Downed Host retry status false with host: intr1n18(192.168.0.18):9160
11:48:38,956 ERROR ~ Could not fullfill request on this host CassandraClient<intr1n19:9160-594>
11:48:38,956 ERROR ~ Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
        at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143)
        at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:159)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:165)
        at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:67)
        at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:63)
        at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
        at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
        at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:62)
    ...
Caused by: TimedOutException()
        at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
        at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732)
        at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:704)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:149)
        ... 23 more
11:48:48,961 ERROR ~ Could not fullfill request on this host CassandraClient<intr1n17:9160-577>
11:48:48,961 ERROR ~ Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
        at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143)
        at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:159)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:165)
        at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:67)
        at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:63)
        at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
        at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
        at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:62)
       ...
Caused by: TimedOutException()
        at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
        at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732)
        at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:704)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:149)
        ... 23 more
11:48:51,025 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route
to host
        at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
        at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
        at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
        at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.verifyConnection(CassandraHostRetryService.java:116)
        at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.run(CassandraHostRetryService.java:96)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
        ... 13 more
11:48:51,025 ERROR ~ Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable
to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route
to host
11:48:51,025 INFO  ~ Downed Host retry status false with host: intr1n18(192.168.0.18):9160



Address         Status State   Load            Owns    Token
                                                       ffffffffffffffff
192.168.0.1     Up     Normal  11.26 GB        5.00%   0cc
192.168.0.2     Up     Normal  11.23 GB        5.00%   199
192.168.0.3     Up     Normal  11.58 GB        5.00%   266
192.168.0.4     Up     Normal  6.77 GB         5.00%   333
192.168.0.5     Up     Normal  6.86 GB         5.00%   400
192.168.0.6     Up     Normal  6.81 GB         5.00%   4cc
192.168.0.7     Up     Normal  6.88 GB         5.00%   599
192.168.0.8     Up     Normal  6.84 GB         5.00%   666
192.168.0.9     Up     Normal  6.52 GB         5.00%   733
192.168.0.10    Up     Normal  5.17 GB         5.00%   7ff
192.168.0.11    Up     Normal  6.75 GB         5.00%   8cc
192.168.0.12    Up     Normal  7.06 GB         5.00%   999
192.168.0.13    Up     Normal  7.27 GB         5.00%   a66
192.168.0.14    Up     Normal  7.71 GB         5.00%   b33
192.168.0.15    Up     Normal  7.46 GB         5.00%   c00
192.168.0.16    Up     Normal  6.94 GB         5.00%   ccc
192.168.0.17    Up     Normal  6.45 GB         5.00%   d99
192.168.0.18    Down   Normal  ?               5.00%   e66
192.168.0.19    Up     Normal  6.26 GB         5.00%   f33
192.168.0.20    Up     Normal  6.33 GB         5.00%   ffffffffffffffff



> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2081
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: linux, hector + cassandra
>            Reporter: Thibaut
>            Priority: Blocker
>             Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level
ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar),
I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar
and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160
host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException:
No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status
false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on
this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra
cli.
> 192.168.0.1     Up     Normal  8.02 GB         5.00%   0cc
> 192.168.0.2     Up     Normal  7.96 GB         5.00%   199
> 192.168.0.3     Up     Normal  8.24 GB         5.00%   266
> 192.168.0.4     Up     Normal  4.94 GB         5.00%   333
> 192.168.0.5     Up     Normal  5.02 GB         5.00%   400
> 192.168.0.6     Up     Normal  5 GB            5.00%   4cc
> 192.168.0.7     Up     Normal  5.1 GB          5.00%   599
> 192.168.0.8     Up     Normal  5.07 GB         5.00%   666
> 192.168.0.9     Up     Normal  4.78 GB         5.00%   733
> 192.168.0.10    Up     Normal  4.34 GB         5.00%   7ff
> 192.168.0.11    Up     Normal  5.01 GB         5.00%   8cc
> 192.168.0.12    Up     Normal  5.31 GB         5.00%   999
> 192.168.0.13    Up     Normal  5.56 GB         5.00%   a66
> 192.168.0.14    Up     Normal  5.82 GB         5.00%   b33
> 192.168.0.15    Up     Normal  5.57 GB         5.00%   c00
> 192.168.0.16    Up     Normal  5.03 GB         5.00%   ccc
> 192.168.0.17    Up     Normal  4.77 GB         5.00%   d99
> 192.168.0.18    Down   Normal  ?               5.00%   e66
> 192.168.0.19    Up     Normal  4.78 GB         5.00%   f33
> 192.168.0.20    Up     Normal  4.83 GB         5.00%   ffffffffffffffff

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message