incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mcasandra <mohitanch...@gmail.com>
Subject All nodes down even though ring shows up
Date Thu, 14 Apr 2011 19:45:53 GMT
I ran stress test to read 50K rows and since then I am getting below error
even though ring show all nodes are up:

ERROR 12:40:29,999 Exception:
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked
down. Retry burden pushed out to client.
        at
me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:308)
        at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:213)
        at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
        at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:100)
        at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:106)
        at
me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:203)
        at
me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:200)
        at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
        at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
        at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:200)
        at
com.riptano.cassandra.stress.InsertCommand.call(InsertCommand.java:117)
        at
com.riptano.cassandra.stress.InsertCommand.call(InsertCommand.java:1)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)


---

No errors logged in the system.log and tpsats shows nothing.

nodetool -h dsdb1 tpstats
Pool Name                    Active   Pending      Completed
ReadStage                         0         0          50176
RequestResponseStage              0         0         207223
MutationStage                     0         0         199473
ReadRepairStage                   0         0          14615
GossipStage                       0         0          39835
AntiEntropyStage                  0         0              0
MigrationStage                    0         0            207
MemtablePostFlusher               0         0            386
StreamStage                       0         0              0
FlushWriter                       0         0            385
FILEUTILS-DELETE-POOL             0         0           1446
MiscStage                         0         0              0
FlushSorter                       0         0              0
InternalResponseStage             0         0           1230
HintedHandoff                     0         0              7


compaction stats say pending: 0



--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/All-nodes-down-even-though-ring-shows-up-tp6274152p6274152.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Mime
View raw message