cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jian Fang <jian.fang.subscr...@gmail.com>
Subject Re: Cassandra timeout exception when works with hadoop
Date Thu, 28 Jul 2011 20:19:00 GMT
I did not set the consistency level because I didn't find this option in the
ConfigHelper class. I guess it should use level one by default.

Actually, I only twisted the word count example a bit. Here is the code
snippet,

        getConf().set(CONF_COLUMN_NAME, columnName);

        Job job = new Job(getConf(), KEYSPACE);
        job.setJarByClass(WorkIdFinder.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(ReducerToFilesystem.class);
        job.setReducerClass(ReducerToFilesystem.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileOutputFormat.setOutputPath(job, new Path(OUTPUT_PATH_PREFIX +
columnFamily));

        job.setInputFormatClass(ColumnFamilyInputFormat.class);

        ConfigHelper.setRpcPort(job.getConfiguration(), "9260");
        ConfigHelper.setInitialAddress(job.getConfiguration(),
"dnjsrcha01");
        ConfigHelper.setPartitioner(job.getConfiguration(),
"org.apache.cassandra.dht.RandomPartitioner");
        ConfigHelper.setInputColumnFamily(job.getConfiguration(), KEYSPACE,
columnFamily);
        ConfigHelper.setRangeBatchSize(job.getConfiguration(), batchSize);
        SlicePredicate predicate = new
SlicePredicate().setColumn_names(Arrays.asList(ByteBufferUtil.bytes(columnName)));
        ConfigHelper.setInputSlicePredicate(job.getConfiguration(),
predicate);

Yes, I have one task tracker running on each Cassandra node.

Thanks,

John


On Thu, Jul 28, 2011 at 3:51 PM, Jeremy Hanna <jeremy.hanna1234@gmail.com>wrote:

> Just wondering - what consistency level are you using for hadoop reads?
>  Also, do you have task trackers running on the cassandra nodes so that
> reads will be local?
>
> On Jul 28, 2011, at 2:46 PM, Jian Fang wrote:
>
> > I changed the rpc_timeout_in_ms to 30000 and 40000, then changed the
> cassandra.range.batch.size from 4096 to 1024,
> > but still 40% tasks got timeout exceptions.
> >
> > Not sure if this is caused by Cassandra speed performance (8G heap size
> for about 100G of data) or the way how the Cassandra-hadoop integration
> > is implemented. I rarely saw any timeout exceptions when I use hector to
> get back data.
> >
> > Thanks,
> >
> > John
> >
> > On Thu, Jul 28, 2011 at 12:45 PM, Jian Fang <
> jian.fang.subscribe@gmail.com> wrote:
> >
> > My current setting is 10000. I will try 30000.
> >
> > Thanks,
> >
> > John
> >
> > On Thu, Jul 28, 2011 at 12:39 PM, Jeremy Hanna <
> jeremy.hanna1234@gmail.com> wrote:
> > See http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting - I
> would probably start with setting your rpc_timeout_in_ms to something like
> 30000.
> >
> > On Jul 28, 2011, at 11:09 AM, Jian Fang wrote:
> >
> > > Hi,
> > >
> > > I run Cassandra 0.8.2 and hadoop 0.20.2 on three nodes, each node
> includes a Cassandra instance and a hadoop data node.
> > > I created a simple hadoop job to scan a Cassandra column value in a
> column family and write it to a file system if it meets some conditions.
> > > I keep getting the following timeout exceptions. Is this caused by my
> settings in Cassandra? Or how could I change the timeout value on the
> > > Cassandra Hadoop API to get around this problem?
> > >
> > >
> > > 11/07/28 12:02:47 INFO mapred.JobClient: Task Id :
> attempt_201107281151_0001_m_000052_0, Status : FAILED
> > > java.lang.RuntimeException: TimedOutException()
> > >     at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:265)
> > >     at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:279)
> > >     at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
> > >     at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> > >     at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> > >     at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
> > >     at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
> > >     at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> > >     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> > >     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> > >     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > >     at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > Caused by: TimedOutException()
> > >     at
> org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12590)
> > >     at
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:762)
> > >     at
> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
> > >     at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:243)
> > >     ... 11 more
> > >
> > > Thanks in advance,
> > >
> > > John
> >
> >
> >
>
>

Mime
View raw message