cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Help needed to solve an issue with Cassandra cluster
Date Mon, 20 Feb 2012 20:25:11 GMT
If you are starting out I recommend using at least 0.8, the memory management will make you
life so much easier. 

if the schema is not fully replicated solve that problem first. Using cassandra-cli and run
the describe cluster command. If there are multiple schemas listed nuke it and start again
(assuming you can nuke it.)

Not sure about creating schema with Hector. But whatever the method be careful to only run
one schema update at a time. Personally I prefer to create schema using the CLI or the CQL
( in 1.0). 

> Caused by: TimedOutException()
> 	at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
> 	at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
> 	at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)

Happens because less than CL nodes replied before rpc_timeout. Check nodetool ring to see
that nodes are up. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/02/2012, at 6:27 AM, Kasun Weranga wrote:

> Hi All,
> 
> I have setup 3 nodes Cassandra cluster(cassandra version is 0.7).  In the code base I
am creating few column families using Hector client with default consistency level and the
replication factor of key space is 2. 
> 
> While running the code I am getting following exception repeatedly on client side. (this
timeout comes when trying to find whether previously written column families are exist or
not)
> 
> [2012-02-20 11:40:52,324]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052255]
time[69] tag[META_READ.success_]
> [2012-02-20 11:40:52,403]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052341]
time[62] tag[READ.fail_]
> [2012-02-20 11:40:52,482]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052405]
time[77] tag[READ.fail_]
> [2012-02-20 11:40:52,932]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052483]
time[449] tag[META_WRITE.success_]
> [2012-02-20 11:40:52,935]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756052933]
time[2] tag[READ.fail_]
> [2012-02-20 11:40:53,037]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756053035]
time[2] tag[READ.fail_]
> [2012-02-20 11:41:03,173] ERROR {me.prettyprint.cassandra.connection.HConnectionManager}
-  Could not fullfill request on this host CassandraClient<192.168.0.101:9160-19>
> [2012-02-20 11:41:03,176] ERROR {me.prettyprint.cassandra.connection.HConnectionManager}
-  Exception: 
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> 	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
> 	at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
> 	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
> 	at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
> 	at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:125)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
> 	at org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: TimedOutException()
> 	at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
> 	at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
> 	at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
> 	... 24 more
> [2012-02-20 11:41:03,181]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756053139]
time[10042] tag[READ.fail_]
> [2012-02-20 11:41:03,208]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756053139]
time[10068] tag[READ.success_]
> [2012-02-20 11:41:03,239]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063214]
time[25] tag[READ.success_]
> [2012-02-20 11:41:03,288]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063252]
time[36] tag[WRITE.success_]
> [2012-02-20 11:41:03,322]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063289]
time[33] tag[WRITE.success_]
> [2012-02-20 11:41:03,325]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063322]
time[3] tag[READ.fail_]
> [2012-02-20 11:41:03,330]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063326]
time[4] tag[READ.fail_]
> [2012-02-20 11:41:03,629]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063331]
time[298] tag[META_WRITE.success_]
> [2012-02-20 11:41:03,632]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063629]
time[3] tag[READ.fail_]
> [2012-02-20 11:41:03,735]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063732]
time[3] tag[READ.fail_]
> [2012-02-20 11:41:13,842] ERROR {me.prettyprint.cassandra.connection.HConnectionManager}
-  Could not fullfill request on this host CassandraClient<192.168.0.100:9160-6>
> [2012-02-20 11:41:13,842] ERROR {me.prettyprint.cassandra.connection.HConnectionManager}
-  Exception: 
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> 	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
> 	at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
> 	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
> 	at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
> 	at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:153)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
> 	at org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: TimedOutException()
> 	at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
> 	at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
> 	at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
> 	... 24 more
> [2012-02-20 11:41:13,846]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063836]
time[10010] tag[READ.fail_]
> [2012-02-20 11:41:13,854]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756063836]
time[10018] tag[READ.success_]
> [2012-02-20 11:41:13,861]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073855]
time[6] tag[READ.success_]
> [2012-02-20 11:41:13,867]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073862]
time[5] tag[WRITE.success_]
> [2012-02-20 11:41:13,905]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073867]
time[38] tag[WRITE.success_]
> [2012-02-20 11:41:13,908]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073906]
time[2] tag[READ.fail_]
> [2012-02-20 11:41:13,911]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073909]
time[2] tag[READ.fail_]
> [2012-02-20 11:41:14,835]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756073912]
time[923] tag[META_WRITE.success_]
> [2012-02-20 11:41:14,838]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756074836]
time[2] tag[READ.fail_]
> [2012-02-20 11:41:15,372]  INFO {me.prettyprint.cassandra.hector.TimingLogger} -  start[1329756074939]
time[433] tag[READ.fail_]
> [2012-02-20 11:41:25,480] ERROR {me.prettyprint.cassandra.connection.HConnectionManager}
-  Could not fullfill request on this host CassandraClient<192.168.0.102:9160-41>
> [2012-02-20 11:41:25,481] ERROR {me.prettyprint.cassandra.connection.HConnectionManager}
-  Exception: 
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> 	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
> 	at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
> 	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:156)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSlice(KeyspaceServiceImpl.java:401)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:67)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery$1.doInKeyspace(ThriftMultigetSliceQuery.java:59)
> 	at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
> 	at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:72)
> 	at me.prettyprint.cassandra.model.thrift.ThriftMultigetSliceQuery.execute(ThriftMultigetSliceQuery.java:58)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.isTableExists(CassandraDataStore.java:509)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.createTable(CassandraDataStore.java:550)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraDataStore.initialize(CassandraDataStore.java:180)
> 	at org.wso2.carbon.bam.core.persistence.cassandra.CassandraStoreFactory.getDataStore(CassandraStoreFactory.java:88)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStore(PersistenceManager.java:334)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.getDataStoreForTable(PersistenceManager.java:295)
> 	at org.wso2.carbon.bam.core.persistence.PersistenceManager.storeEvent(PersistenceManager.java:54)
> 	at org.wso2.carbon.bam.receiver.internal.QueueWorker.run(QueueWorker.java:63)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: TimedOutException()
> 	at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
> 	at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
> 	at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
> 	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
> 	... 24 more
> 
> 
> 
> Also I can see only few column families has been created in cluster nodes.  In addition
to that those created column families are not identical across the cluster nodes (One node
has META_INFO column family while other nodes doesn't have it ).
> I can see the following types of exceptions printed in two cassandra nodes.
> 
> [2012-02-20 22:28:26,153] ERROR {org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor}
-  Error in ThreadPoolExecutor
> java.lang.IllegalArgumentException: Unknown table/cf pair (BAMKeySpace.META_INFO)
> 	at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148)
> 	at org.apache.cassandra.db.Table.getRow(Table.java:383)
> 	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> 	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
> 	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> [2012-02-20 22:28:26,156] ERROR {org.apache.cassandra.service.AbstractCassandraDaemon}
-  Fatal exception in thread Thread[ReadStage:106,5,main]
> java.lang.IllegalArgumentException: Unknown table/cf pair (BAMKeySpace.META_INFO)
> 	at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:148)
> 	at org.apache.cassandra.db.Table.getRow(Table.java:383)
> 	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> 	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
> 	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> 
> 
> What can be the issue?
> 
> Appreciate your help to resolve this issue.
> 
> 
> Thanks,
> Kasun.
> 
> 
> 
>  
> 


Mime
View raw message