Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of
 brian.jeltema@digitalenvoy.net designates 68.64.43.136 as permitted sender)
From: Brian Jeltema <brian.jeltema@digitalenvoy.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Hadoop/Cassandra 1.2 timeouts
Date: Mon, 24 Jun 2013 12:10:13 -0400
Message-Id: <D1F0CCF4-68DE-4977-8310-DF6E229A4178@digitalenvoy.net>
To: user@cassandra.apache.org
Mime-Version: 1.0 (Apple Message framework v1278)

I'm having problems with Hadoop job failures on a Cassandra 1.2 cluster =
due to=20

    Caused by: TimedOutException()
    2013-06-24 11:29:11,953  INFO  Driver  - 	at =
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassand=
ra.java:12932)

This is running on a 6-node cluster, RF=3D3. If I run the job with =
CL=3DONE, it usually runs pretty well, with an occasional timeout. But
if I run at CL=3DQUORUM, the number of timeouts is often enough to kill =
the job. The table being read is effectively read-only when this job =
runs.
It has from 5 to 10 million rows, with each row having no more than 256 =
columns. Each column typically only has a few hundred bytes of data at =
most.

I've fiddled with the batch-range size and increasing the timeout, =
without a lot of luck. I see some evidence of GC activity in the =
Cassandra logs, but
it's hard to see a clear correlation with the timeouts.

I could use some suggestions on an approach to pin down the root cause.

TIA

Brian=