incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohica Jasha <mohica.ja...@gmail.com>
Subject Re: error in cassandra log file under stress
Date Mon, 05 May 2014 21:17:13 GMT
Most likely it is the following query:

DELETE from conditional_update_lock where resource_id = '${myresourceid}' IF
lock_id = ${myuuid};

We are using datastax 2.0.1 and the datastax threw a ReadtimeoutException





On Mon, May 5, 2014 at 5:03 PM, Mohica Jasha <mohica.jasha@gmail.com> wrote:

> forgot to say which Cassandra version we are using.
>
> Our prod deployment:
>
> C* 2.0.5, DC1:3, DC2:3
>
>
>
>
> On Mon, May 5, 2014 at 5:01 PM, Mohica Jasha <mohica.jasha@gmail.com>wrote:
>
>> one query in our prod system under heavy load failed.
>> I cant tell which query it was. But I don't think we were using an
>> invalid consistency level since our configuration work all the time except
>> when it goes under heavy load.
>> If I can reproduce this I will pass more information.
>>
>> I wonder if this is due to a race condition in Paxos implementation.
>> I found the following in the cassandra's log file:
>>
>>
>> ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>>>> QueryMessage.java (line 131) Unexpected error during query
>>>
>>> java.lang.UnsupportedOperationException: Invalid consistency level:
>>>> LOCAL_SERIAL
>>>
>>>         at
>>>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>>>
>>>         at
>>>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>>>
>>>         at
>>>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>>>
>>>         at
>>>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>>>
>>>         at
>>>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>>>
>>>         at
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>>
>>>         at
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>
>>>         at
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>>
>>>         at
>>>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>>>
>>>         at
>>>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>>>
>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>
>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>
>>>         at java.lang.Thread.run(Thread.java:744)
>>>
>>> ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794
>>>> ErrorMessage.java (line 222) Unexpected exception during request
>>>
>>> java.lang.UnsupportedOperationException: Invalid consistency level:
>>>> LOCAL_SERIAL
>>>
>>>         at
>>>> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137)
>>>
>>>         at
>>>> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416)
>>>
>>>         at
>>>> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
>>>
>>>         at
>>>> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
>>>
>>>         at
>>>> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
>>>
>>>         at
>>>> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304)
>>>
>>>         at
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>>
>>>         at
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>
>>>         at
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>>
>>>         at
>>>> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>>>
>>>         at
>>>> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>>>
>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>
>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>
>>>         at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>

Mime
View raw message