cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hanna <jeremy.hanna1...@gmail.com>
Subject Re: Cassandra timeout exception when works with hadoop
Date Thu, 28 Jul 2011 19:51:47 GMT
Just wondering - what consistency level are you using for hadoop reads?  Also, do you have
task trackers running on the cassandra nodes so that reads will be local?

On Jul 28, 2011, at 2:46 PM, Jian Fang wrote:

> I changed the rpc_timeout_in_ms to 30000 and 40000, then changed the cassandra.range.batch.size
from 4096 to 1024,
> but still 40% tasks got timeout exceptions.
> 
> Not sure if this is caused by Cassandra speed performance (8G heap size for about 100G
of data) or the way how the Cassandra-hadoop integration
> is implemented. I rarely saw any timeout exceptions when I use hector to get back data.

> 
> Thanks,
> 
> John
>  
> On Thu, Jul 28, 2011 at 12:45 PM, Jian Fang <jian.fang.subscribe@gmail.com> wrote:
> 
> My current setting is 10000. I will try 30000. 
> 
> Thanks,
> 
> John
> 
> On Thu, Jul 28, 2011 at 12:39 PM, Jeremy Hanna <jeremy.hanna1234@gmail.com> wrote:
> See http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting - I would probably
start with setting your rpc_timeout_in_ms to something like 30000.
> 
> On Jul 28, 2011, at 11:09 AM, Jian Fang wrote:
> 
> > Hi,
> >
> > I run Cassandra 0.8.2 and hadoop 0.20.2 on three nodes, each node includes a Cassandra
instance and a hadoop data node.
> > I created a simple hadoop job to scan a Cassandra column value in a column family
and write it to a file system if it meets some conditions.
> > I keep getting the following timeout exceptions. Is this caused by my settings in
Cassandra? Or how could I change the timeout value on the
> > Cassandra Hadoop API to get around this problem?
> >
> >
> > 11/07/28 12:02:47 INFO mapred.JobClient: Task Id : attempt_201107281151_0001_m_000052_0,
Status : FAILED
> > java.lang.RuntimeException: TimedOutException()
> >     at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:265)
> >     at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:279)
> >     at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
> >     at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> >     at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> >     at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
> >     at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
> >     at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> >     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >     at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: TimedOutException()
> >     at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12590)
> >     at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:762)
> >     at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
> >     at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:243)
> >     ... 11 more
> >
> > Thanks in advance,
> >
> > John
> 
> 
> 


Mime
View raw message