Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	thelastpickle.com; b=00cfwc9tjenNOBlnzH/RFWsFws1RS86njAExvYRAOgk
	DG44TClh7n5FIQE4tyC1A5xJozR0iSbqEamuFaHWhu/lwqKkRW87dNs9PPHxpIei
	byrgy/3pVa0wOLOkdvOme2dJhz0XVuycUryl3poadlPHPdjsWeFkwx4D1IxWdejQ
	=
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1257)
Subject: Re: Request timeout and host marked down
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: 
 <CAGiE6h99Fa0zMarQ6jex4s+MUX4eEyUwT17JEbGDdcJnR2Zrzg@mail.gmail.com>
Date: Mon, 9 Apr 2012 09:15:59 +1200
Content-Transfer-Encoding: quoted-printable
Message-Id: <16F5BF0E-9C56-4F23-AD1E-910AEDF72DF1@thelastpickle.com>
References: 
 <CAGiE6h99Fa0zMarQ6jex4s+MUX4eEyUwT17JEbGDdcJnR2Zrzg@mail.gmail.com>
To: user@cassandra.apache.org

You need to see if the timeout is from the client to the server, or =
between the server nodes.=20

If it's server side a TimedOutException will be thrown from thrift. Take =
a look at the nodetool tpstats on the servers, you will probably see =
lots of "Pending" tasks. Basically the cluster is overloaded. Consider:

* check the IO, CPU, GC state on the servers.=20
* ensuring the data and requests are evenly spread around the cluster.=20=

* reducing the number of columns read in a select.=20

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 6/04/2012, at 5:30 AM, Daning Wang wrote:

> Hi all,
>=20
> We are using Hector and ofter we see lots of timeout exception in the =
log, I know that the hector can failover to other node, but I want to =
reduce the number of timeouts.
>=20
> any hector parameter I should change to reduce this error?
>=20
> also, on the server side, any kind of tunning need to do for the =
timeout?
> =20
>=20
> Thanks in advance.
>=20
>=20
> 12/04/04 15:13:20 ERROR =
com.netseer.services.keywordstat.io.KeywordServiceImpl: Timout 10000 ms
> 12/04/04 15:13:25 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: MARK HOST AS =
DOWN TRIGGERED for host 10.28.78.123(10.28.78.123):9160
> 12/04/04 15:13:25 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: Pool state on =
shutdown: =
<ConcurrentCassandraClientPoolByHost>:{10.28.78.123(10.28.78.123):9160}; =
IsActive?: true; Active: 1; Blocked: 0; Idle: 5; NumBeforeExhausted: 19
> 12/04/04 15:13:44 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: MARK HOST AS =
DOWN TRIGGERED for host 10.240.113.171(10.240.113.171):9160
> 12/04/04 15:13:44 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: Pool state on =
shutdown: =
<ConcurrentCassandraClientPoolByHost>:{10.240.113.171(10.240.113.171):9160=
}; IsActive?: true; Active: 1; Blocked: 0; Idle: 5; NumBeforeExhausted: =
19
> 12/04/04 15:13:46 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: MARK HOST AS =
DOWN TRIGGERED for host 10.28.78.123(10.28.78.123):9160
> 12/04/04 15:13:46 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: Pool state on =
shutdown: =
<ConcurrentCassandraClientPoolByHost>:{10.28.78.123(10.28.78.123):9160}; =
IsActive?: true; Active: 1; Blocked: 0; Idle: 5; NumBeforeExhausted: 19
> 12/04/04 15:13:46 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: MARK HOST AS =
DOWN TRIGGERED for host 10.123.83.114(10.123.83.114):9160
> 12/04/04 15:13:46 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: Pool state on =
shutdown: =
<ConcurrentCassandraClientPoolByHost>:{10.123.83.114(10.123.83.114):9160};=
 IsActive?: true; Active: 1; Blocked: 0; Idle: 5; NumBeforeExhausted: 19
> 12/04/04 15:13:46 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: MARK HOST AS =
DOWN TRIGGERED for host 10.6.115.239(10.6.115.239):9160
> 12/04/04 15:13:46 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: Pool state on =
shutdown: =
<ConcurrentCassandraClientPoolByHost>:{10.6.115.239(10.6.115.239):9160}; =
IsActive?: true; Active: 1; Blocked: 0; Idle: 5; NumBeforeExhausted: 19
> 12/04/04 15:13:49 ERROR =
com.netseer.services.keywordstat.io.KeywordServiceImpl: Timout 10000 ms
> 12/04/04 15:13:49 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: MARK HOST AS =
DOWN TRIGGERED for host 10.120.205.48(10.120.205.48):9160
> 12/04/04 15:13:49 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: Pool state on =
shutdown: =
<ConcurrentCassandraClientPoolByHost>:{10.120.205.48(10.120.205.48):9160};=
 IsActive?: true; Active: 3; Blocked: 0; Idle: 3; NumBeforeExhausted: 17
> 12/04/04 15:13:50 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: MARK HOST AS =
DOWN TRIGGERED for host 10.28.20.200(10.28.20.200):9160
> 12/04/04 15:13:50 ERROR =
me.prettyprint.cassandra.connection.HConnectionManager: Pool state on =
shutdown: =
<ConcurrentCassandraClientPoolByHost>:{10.28.20.200(10.28.20.200):9160}; =
IsActive?: true; Active: 2; Blocked: 0; Idle: 4; NumBeforeExhausted: 18
> 12/04/04 15:13:51 ERROR =
com.netseer.services.keywordstat.io.KeywordServiceImpl: Timout 10000 ms