cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: TimedOutException and UnavailableException from multiGetSliceQuery
Date Wed, 05 Oct 2011 20:43:43 GMT
5000 rows in a mutli get is way, way, way (did I say way ? ) to many. 

Whenever you get a TimedOutException check the tp stats on the nodes, you will normally see
a high pending count. Every row get get turns into an message in a TP. So if you ask for 5k
rows you flood the TP with 5k messages which will often result in the node(s) been temporarily
overloaded. 

More is not always more. I would guess 100 as a starting point, i would be doubtful that you
would see much benefit beyond 1000. pycassa defaults to 1024 https://github.com/pycassa/pycassa/blob/master/pycassa/columnfamily.py#L63


 Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 6/10/2011, at 9:14 AM, Yuhan Zhang wrote:

> Hi all,
> 
> I have been experiencing the unavailableException and TimedOutException on a 3-node cassandra
cluster
> during a multiGetSliceQuery with 1000 columns. Since there are many keys involved in
the query, I divided
> them into groups of 5000 rows and process each group individually in a for loop. but
seems like it is not helping..
> Once the TimedOutException appears, further requests to cassandra will cause UnavailableException.
> However, the servers can recover after a while without intervention. 
> 
> Which settings should I pay attention to in order to fix the problem? This problem becomes
very frequent recently.
>  
> 
> Thank you.
> 
> Yuhan
> 
> The exception looks like:
> 
> 1/10/05 13:05:31 ERROR connection.HConnectionManager: Could not fullfill request on this
host CassandraClient<ec2-75-101-238-70.compute-1.amazonaws.com:9160-33>
> 11/10/05 13:05:31 ERROR connection.HConnectionManager: Exception: 
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
>     at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
>     at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161)
>     at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143)
>     at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
>     at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:155)
> ...
> Caused by: TimedOutException()
>     at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
>     at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732)
> 
> 
> 11/10/05 20:06:05 ERROR connection.HConnectionManager: Could not fullfill request on
this host CassandraClient<ec2-184-73-116-237.compute-1.amazonaws.com:9160-15>
> 11/10/05 20:06:05 ERROR connection.HConnectionManager: Exception: 
> me.prettyprint.hector.api.exceptions.HUnavailableException: UnavailableException()
>     at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:50)
>     at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397)
>     at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383)
>     at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
> 
> Caused by: UnavailableException()
>     at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9620)
>     at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
>     at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
>     at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
>     ... 35 more
> 


Mime
View raw message