cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kasun Weranga <kas...@wso2.com>
Subject Help needed to solve an issue with Cassandra cluster
Date Mon, 20 Feb 2012 17:27:27 GMT
Hi All,

I have setup 3 nodes Cassandra cluster(cassandra version is 0.7).  In the
code base I am creating few column families using Hector client with
default consistency level and the replication factor of key space is 2.

While running the code I am getting following exception repeatedly on
client side. (this timeout comes when trying to find whether previously
written column families are exist or not)

[2012-02-20 11:40:52,324]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052255]
time[69] tag[META_READ.success_]
[2012-02-20 11:40:52,403]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052341]
time[62] tag[READ.fail_]
[2012-02-20 11:40:52,482]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052405]
time[77] tag[READ.fail_]
[2012-02-20 11:40:52,932]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052483]
time[449] tag[META_WRITE.success_]
[2012-02-20 11:40:52,935]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052933]
time[2] tag[READ.fail_]
[2012-02-20 11:40:53,037]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756053035]
time[2] tag[READ.fail_]
[2012-02-20 11:41:03,173] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} -  Could not
fullfill request on this host CassandraClient<192.168.0.101:9160-19>
[2012-02-20 11:41:03,176] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} -  Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
 at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
 at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
 at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
 at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
 at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:125)
 at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
 at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
 at
org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
 at
org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
at
org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
... 24 more
[2012-02-20 11:41:03,181]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756053139]
time[10042] tag[READ.fail_]
[2012-02-20 11:41:03,208]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756053139]
time[10068] tag[READ.success_]
[2012-02-20 11:41:03,239]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063214]
time[25] tag[READ.success_]
[2012-02-20 11:41:03,288]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063252]
time[36] tag[WRITE.success_]
[2012-02-20 11:41:03,322]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063289]
time[33] tag[WRITE.success_]
[2012-02-20 11:41:03,325]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063322]
time[3] tag[READ.fail_]
[2012-02-20 11:41:03,330]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063326]
time[4] tag[READ.fail_]
[2012-02-20 11:41:03,629]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063331]
time[298] tag[META_WRITE.success_]
[2012-02-20 11:41:03,632]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063629]
time[3] tag[READ.fail_]
[2012-02-20 11:41:03,735]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063732]
time[3] tag[READ.fail_]
[2012-02-20 11:41:13,842] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} -  Could not
fullfill request on this host CassandraClient<192.168.0.100:9160-6>
[2012-02-20 11:41:13,842] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} -  Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
 at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
 at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
 at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
 at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
 at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:153)
 at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
 at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
 at
org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
 at
org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
at
org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
... 24 more
[2012-02-20 11:41:13,846]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063836]
time[10010] tag[READ.fail_]
[2012-02-20 11:41:13,854]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063836]
time[10018] tag[READ.success_]
[2012-02-20 11:41:13,861]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073855]
time[6] tag[READ.success_]
[2012-02-20 11:41:13,867]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073862]
time[5] tag[WRITE.success_]
[2012-02-20 11:41:13,905]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073867]
time[38] tag[WRITE.success_]
[2012-02-20 11:41:13,908]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073906]
time[2] tag[READ.fail_]
[2012-02-20 11:41:13,911]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073909]
time[2] tag[READ.fail_]
[2012-02-20 11:41:14,835]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073912]
time[923] tag[META_WRITE.success_]
[2012-02-20 11:41:14,838]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756074836]
time[2] tag[READ.fail_]
[2012-02-20 11:41:15,372]  INFO
{me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756074939]
time[433] tag[READ.fail_]
[2012-02-20 11:41:25,480] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} -  Could not
fullfill request on this host CassandraClient<192.168.0.102:9160-41>
[2012-02-20 11:41:25,481] ERROR
{me.prettyprint.cassandra.connection.HConnectionManager} -  Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
 at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
 at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
 at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
 at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
 at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:180)
 at
org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
 at
org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
at
org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
 at
org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
 at
org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
at
org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
 at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
... 24 more



Also I can see only few column families has been created in cluster
nodes.  In addition to that those created column families are not
identical across the cluster nodes (One node has
META_INFO<https://192.168.0.101:9448/carbon/cassandramgt/keyspace_dashboard.jsp?name=BAMKeySpace&loginStatus=true#>
column
family while other nodes doesn't have it ).
I can see the following types of exceptions printed in two cassandra nodes.

[2012-02-20 22:28:26,153] ERROR
{org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor} -  Error in
ThreadPoolExecutor
java.lang.IllegalArgumentException: Unknown table/cf pair
(BAMKeySpace.META_INFO)
at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148)
 at org.apache.cassandra.db.Table.getRow(Table.java:383)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
 at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
[2012-02-20 22:28:26,156] ERROR
{org.apache.cassandra.service.AbstractCassandraDaemon} -  Fatal exception
in thread Thread[ReadStage:106,5,main]
java.lang.IllegalArgumentException: Unknown table/cf pair
(BAMKeySpace.META_INFO)
at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148)
 at org.apache.cassandra.db.Table.getRow(Table.java:383)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
 at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)


What can be the issue?

Appreciate your help to resolve this issue.


Thanks,
Kasun.

Mime
View raw message