Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 422E89DE6 for ; Wed, 5 Oct 2011 20:15:19 +0000 (UTC) Received: (qmail 40249 invoked by uid 500); 5 Oct 2011 20:15:17 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 40223 invoked by uid 500); 5 Oct 2011 20:15:17 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 40215 invoked by uid 99); 5 Oct 2011 20:15:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Oct 2011 20:15:17 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [74.125.149.207] (HELO na3sys009aog112.obsmtp.com) (74.125.149.207) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 05 Oct 2011 20:15:11 +0000 Received: from mail-vw0-f42.google.com ([209.85.212.42]) (using TLSv1) by na3sys009aob112.postini.com ([74.125.148.12]) with SMTP; Wed, 05 Oct 2011 13:14:50 PDT Received: by mail-vw0-f42.google.com with SMTP id 1so1972200vwl.15 for ; Wed, 05 Oct 2011 13:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=onescreen.com; s=google; h=mime-version:date:message-id:subject:from:to:content-type; bh=+Yfm4i17nfmz5NAbV33sX6UzvYS2GfXq+xkiMf8WK00=; b=lYxPlkalQHU5G+ovKt17EjQG089iuMvO28MRyLWx0Fvxm9lVAjz/V0LrtbKcvdAYC3 1v2DNdcUpcQzMqPFAkZqgL8ZXsBs025Gpn/oKz3YgUm+wkdZuojhSDPUZEwQkYegbIQE v5eXyhfrASX6t5Te63kgJpseqLtkYWSuWCo4o= MIME-Version: 1.0 Received: by 10.52.23.17 with SMTP id i17mr2820280vdf.353.1317845688728; Wed, 05 Oct 2011 13:14:48 -0700 (PDT) Received: by 10.52.33.78 with HTTP; Wed, 5 Oct 2011 13:14:48 -0700 (PDT) Date: Wed, 5 Oct 2011 13:14:48 -0700 Message-ID: Subject: TimedOutException and UnavailableException from multiGetSliceQuery From: Yuhan Zhang To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf307d05ec253a7204ae92dc33 --20cf307d05ec253a7204ae92dc33 Content-Type: text/plain; charset=ISO-8859-1 Hi all, I have been experiencing the unavailableException and TimedOutException on a 3-node cassandra cluster during a multiGetSliceQuery with 1000 columns. Since there are many keys involved in the query, I divided them into groups of 5000 rows and process each group individually in a for loop. but seems like it is not helping.. Once the TimedOutException appears, further requests to cassandra will cause UnavailableException. However, the servers can recover after a while without intervention. Which settings should I pay attention to in order to fix the problem? This problem becomes very frequent recently. Thank you. Yuhan The exception looks like: 1/10/05 13:05:31 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient 11/10/05 13:05:31 ERROR connection.HConnectionManager: Exception: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143) at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:155) ... Caused by: TimedOutException() at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104) at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732) 11/10/05 20:06:05 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient 11/10/05 20:06:05 ERROR connection.HConnectionManager: Exception: me.prettyprint.hector.api.exceptions.HUnavailableException: UnavailableException() at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:50) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:397) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:383) at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101) Caused by: UnavailableException() at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9620) at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636) at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388) ... 35 more --20cf307d05ec253a7204ae92dc33 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi all,

I have been experiencing the unavailableException and TimedO= utException on a 3-node cassandra cluster
during a multiGetSliceQuery wi= th 1000 columns. Since there are many keys involved in the query, I divided=
them into groups of 5000 rows and process each group individually in a for = loop. but seems like it is not helping..
Once the TimedOutException appe= ars, further requests to cassandra will cause UnavailableException.
However, the servers can recover after a while without intervention.
Which settings should I pay attention to in order to fix the problem? Thi= s problem becomes very frequent recently.
=A0

Thank you.

Yuhan

The exception looks like:

1/10/05 13:05:31 ERROR connec= tion.HConnectionManager: Could not fullfill request on this host CassandraC= lient<ec2-75-101-238-70.compute-1.amazonaws.com:9160-33>
11/10/05 = 13:05:31 ERROR connection.HConnectionManager: Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException(= )
=A0=A0=A0 at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl= .translate(ExceptionsTranslatorImpl.java:32)
=A0=A0=A0 at me.prettyprint= .cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:1= 61)
=A0=A0=A0 at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute= (KeyspaceServiceImpl.java:143)
=A0=A0=A0 at me.prettyprint.cassandra.ser= vice.Operation.executeAndSetResult(Operation.java:101)
=A0=A0=A0 at me.p= rettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HCon= nectionManager.java:155)
...
Caused by: TimedOutException()
=A0=A0=A0 at org.apache.cassandra.= thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
=A0= =A0=A0 at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slice= s(Cassandra.java:732)


11/10/05 20:06:05 ERROR connection.HConnectionManager: Could not fu= llfill request on this host CassandraClient<ec2-184-73-116-237.compute-1= .amazonaws.com:9160-15>
11/10/05 20:06:05 ERROR connection.HConnectio= nManager: Exception:
me.prettyprint.hector.api.exceptions.HUnavailableException: UnavailableExce= ption()
=A0=A0=A0 at me.prettyprint.cassandra.service.ExceptionsTranslat= orImpl.translate(ExceptionsTranslatorImpl.java:50)
=A0=A0=A0 at me.prett= yprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl= .java:397)
=A0=A0=A0 at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execut= e(KeyspaceServiceImpl.java:383)
=A0=A0=A0 at me.prettyprint.cassandra.se= rvice.Operation.executeAndSetResult(Operation.java:101)

Caused by: U= navailableException()
=A0=A0=A0 at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.re= ad(Cassandra.java:9620)
=A0=A0=A0 at org.apache.cassandra.thrift.Cassand= ra$Client.recv_multiget_slice(Cassandra.java:636)
=A0=A0=A0 at org.apach= e.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
=A0=A0=A0 at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execut= e(KeyspaceServiceImpl.java:388)
=A0=A0=A0 ... 35 more

--20cf307d05ec253a7204ae92dc33--