Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1C9647272 for ; Thu, 28 Jul 2011 20:19:31 +0000 (UTC) Received: (qmail 23576 invoked by uid 500); 28 Jul 2011 20:19:29 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23524 invoked by uid 500); 28 Jul 2011 20:19:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23516 invoked by uid 99); 28 Jul 2011 20:19:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jul 2011 20:19:28 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jian.fang.subscribe@gmail.com designates 209.85.210.48 as permitted sender) Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jul 2011 20:19:21 +0000 Received: by pzk32 with SMTP id 32so6128335pzk.35 for ; Thu, 28 Jul 2011 13:19:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=w99tNvAt5rTQQH8MT5T+uMhzxqmLaOKONBWU0Q3Buyk=; b=CQi26pV1IejbKxUOylXt/xSKIhYUq/ez0ur/fl+/o1L6V1JUlcmvVgvVuQUBWc/1lH JYyxBkWKl8B6nNKLbIQL5QH+5kN75MAS8/MBn4cYHOgkc4N/VN8dEKj8ezixkkouBXrT LTd5Bw2FjDjNzkWfGe9gBb+pNH0T4ibEdliwo= MIME-Version: 1.0 Received: by 10.68.36.41 with SMTP id n9mr936186pbj.88.1311884340970; Thu, 28 Jul 2011 13:19:00 -0700 (PDT) Received: by 10.68.40.35 with HTTP; Thu, 28 Jul 2011 13:19:00 -0700 (PDT) In-Reply-To: <9B40159A-C39B-4E0C-9983-0093DD98B418@gmail.com> References: <78A0B668-E656-472B-A78C-C6BD9C1DA935@gmail.com> <9B40159A-C39B-4E0C-9983-0093DD98B418@gmail.com> Date: Thu, 28 Jul 2011 16:19:00 -0400 Message-ID: Subject: Re: Cassandra timeout exception when works with hadoop From: Jian Fang To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec51f969f2144fa04a926e045 --bcaec51f969f2144fa04a926e045 Content-Type: text/plain; charset=ISO-8859-1 I did not set the consistency level because I didn't find this option in the ConfigHelper class. I guess it should use level one by default. Actually, I only twisted the word count example a bit. Here is the code snippet, getConf().set(CONF_COLUMN_NAME, columnName); Job job = new Job(getConf(), KEYSPACE); job.setJarByClass(WorkIdFinder.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(ReducerToFilesystem.class); job.setReducerClass(ReducerToFilesystem.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileOutputFormat.setOutputPath(job, new Path(OUTPUT_PATH_PREFIX + columnFamily)); job.setInputFormatClass(ColumnFamilyInputFormat.class); ConfigHelper.setRpcPort(job.getConfiguration(), "9260"); ConfigHelper.setInitialAddress(job.getConfiguration(), "dnjsrcha01"); ConfigHelper.setPartitioner(job.getConfiguration(), "org.apache.cassandra.dht.RandomPartitioner"); ConfigHelper.setInputColumnFamily(job.getConfiguration(), KEYSPACE, columnFamily); ConfigHelper.setRangeBatchSize(job.getConfiguration(), batchSize); SlicePredicate predicate = new SlicePredicate().setColumn_names(Arrays.asList(ByteBufferUtil.bytes(columnName))); ConfigHelper.setInputSlicePredicate(job.getConfiguration(), predicate); Yes, I have one task tracker running on each Cassandra node. Thanks, John On Thu, Jul 28, 2011 at 3:51 PM, Jeremy Hanna wrote: > Just wondering - what consistency level are you using for hadoop reads? > Also, do you have task trackers running on the cassandra nodes so that > reads will be local? > > On Jul 28, 2011, at 2:46 PM, Jian Fang wrote: > > > I changed the rpc_timeout_in_ms to 30000 and 40000, then changed the > cassandra.range.batch.size from 4096 to 1024, > > but still 40% tasks got timeout exceptions. > > > > Not sure if this is caused by Cassandra speed performance (8G heap size > for about 100G of data) or the way how the Cassandra-hadoop integration > > is implemented. I rarely saw any timeout exceptions when I use hector to > get back data. > > > > Thanks, > > > > John > > > > On Thu, Jul 28, 2011 at 12:45 PM, Jian Fang < > jian.fang.subscribe@gmail.com> wrote: > > > > My current setting is 10000. I will try 30000. > > > > Thanks, > > > > John > > > > On Thu, Jul 28, 2011 at 12:39 PM, Jeremy Hanna < > jeremy.hanna1234@gmail.com> wrote: > > See http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting - I > would probably start with setting your rpc_timeout_in_ms to something like > 30000. > > > > On Jul 28, 2011, at 11:09 AM, Jian Fang wrote: > > > > > Hi, > > > > > > I run Cassandra 0.8.2 and hadoop 0.20.2 on three nodes, each node > includes a Cassandra instance and a hadoop data node. > > > I created a simple hadoop job to scan a Cassandra column value in a > column family and write it to a file system if it meets some conditions. > > > I keep getting the following timeout exceptions. Is this caused by my > settings in Cassandra? Or how could I change the timeout value on the > > > Cassandra Hadoop API to get around this problem? > > > > > > > > > 11/07/28 12:02:47 INFO mapred.JobClient: Task Id : > attempt_201107281151_0001_m_000052_0, Status : FAILED > > > java.lang.RuntimeException: TimedOutException() > > > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:265) > > > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:279) > > > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177) > > > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > > > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > > > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136) > > > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423) > > > at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > > > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > > Caused by: TimedOutException() > > > at > org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12590) > > > at > org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:762) > > > at > org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734) > > > at > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:243) > > > ... 11 more > > > > > > Thanks in advance, > > > > > > John > > > > > > > > --bcaec51f969f2144fa04a926e045 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I did not set the consistency level because I didn't find this option i= n the ConfigHelper class. I guess it should use level one by default.
<= br>Actually, I only twisted the word count example a bit. Here is the code = snippet,

=A0=A0=A0=A0=A0=A0=A0 getConf().set(CONF_COLUMN_NAME, columnName);
<= br>=A0=A0=A0=A0=A0=A0=A0 Job job =3D new Job(getConf(), KEYSPACE);
=A0= =A0=A0=A0=A0=A0=A0 job.setJarByClass(WorkIdFinder.class);
=A0=A0=A0=A0= =A0=A0=A0 job.setMapperClass(TokenizerMapper.class);
=A0=A0=A0=A0=A0=A0= =A0 job.setCombinerClass(ReducerToFilesystem.class);
=A0=A0=A0=A0=A0=A0=A0 job.setReducerClass(ReducerToFilesystem.class);
= =A0=A0=A0=A0=A0=A0=A0 job.setOutputKeyClass(Text.class);
=A0=A0=A0=A0=A0= =A0=A0 job.setOutputValueClass(IntWritable.class);
=A0=A0=A0=A0=A0=A0=A0= FileOutputFormat.setOutputPath(job, new Path(OUTPUT_PATH_PREFIX + columnFa= mily));

=A0=A0=A0=A0=A0=A0=A0 job.setInputFormatClass(ColumnFamilyInputFormat.c= lass);

=A0=A0=A0=A0=A0=A0=A0 ConfigHelper.setRpcPort(job.getConfigur= ation(), "9260");
=A0=A0=A0=A0=A0=A0=A0 ConfigHelper.setInitia= lAddress(job.getConfiguration(), "dnjsrcha01");
=A0=A0=A0=A0=A0=A0=A0 ConfigHelper.setPartitioner(job.getConfiguration(), &= quot;org.apache.cassandra.dht.RandomPartitioner");
=A0=A0=A0=A0=A0= =A0=A0 ConfigHelper.setInputColumnFamily(job.getConfiguration(), KEYSPACE, = columnFamily);
=A0=A0=A0=A0=A0=A0=A0 ConfigHelper.setRangeBatchSize(job.= getConfiguration(), batchSize);
=A0=A0=A0=A0=A0=A0=A0 SlicePredicate predicate =3D new SlicePredicate().set= Column_names(Arrays.asList(ByteBufferUtil.bytes(columnName)));
=A0=A0=A0= =A0=A0=A0=A0 ConfigHelper.setInputSlicePredicate(job.getConfiguration(), pr= edicate);

Yes, I have one task tracker running on each Cassandra nod= e.

Thanks,

John


On Thu, Jul 2= 8, 2011 at 3:51 PM, Jeremy Hanna <jeremy.hanna1234@gmail.com> wrote:<= br>
Just wondering - = what consistency level are you using for hadoop reads? =A0Also, do you have= task trackers running on the cassandra nodes so that reads will be local?<= br>

On Jul 28, 2011, at 2:46 PM, Jian Fang wrote:

> I changed the rpc_timeout_in_ms to 30000 and 40000, then changed the c= assandra.range.batch.size from 4096 to 1024,
> but still 40% tasks got timeout exceptions.
>
> Not sure if this is caused by Cassandra speed performance (8G heap siz= e for about 100G of data) or the way how the Cassandra-hadoop integration > is implemented. I rarely saw any timeout exceptions when I use hector = to get back data.
>
> Thanks,
>
> John
>
> On Thu, Jul 28, 2011 at 12:45 PM, Jian Fang <jian.fang.subscribe@gmail.com> wrote:
>
> My current setting is 10000. I will try 30000.
>
> Thanks,
>
> John
>
> On Thu, Jul 28, 2011 at 12:39 PM, Jeremy Hanna <jeremy.hanna1234@gmail.com> wrote:
> See http://wiki.apache.org/cassandra/HadoopSupport#T= roubleshooting - I would probably start with setting your rpc_timeout_i= n_ms to something like 30000.
>
> On Jul 28, 2011, at 11:09 AM, Jian Fang wrote:
>
> > Hi,
> >
> > I run Cassandra 0.8.2 and hadoop 0.20.2 on three nodes, each node= includes a Cassandra instance and a hadoop data node.
> > I created a simple hadoop job to scan a Cassandra column value in= a column family and write it to a file system if it meets some conditions.=
> > I keep getting the following timeout exceptions. Is this caused b= y my settings in Cassandra? Or how could I change the timeout value on the<= br> > > Cassandra Hadoop API to get around this problem?
> >
> >
> > 11/07/28 12:02:47 INFO mapred.JobClient: Task Id : attempt_201107= 281151_0001_m_000052_0, Status : FAILED
> > java.lang.RuntimeException: TimedOutException()
> > =A0 =A0 at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$R= owIterator.maybeInit(ColumnFamilyRecordReader.java:265)
> > =A0 =A0 at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$R= owIterator.computeNext(ColumnFamilyRecordReader.java:279)
> > =A0 =A0 at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$R= owIterator.computeNext(ColumnFamilyRecordReader.java:177)
> > =A0 =A0 at com.google.common.collect.AbstractIterator.tryToComput= eNext(AbstractIterator.java:140)
> > =A0 =A0 at com.google.common.collect.AbstractIterator.hasNext(Abs= tractIterator.java:135)
> > =A0 =A0 at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.n= extKeyValue(ColumnFamilyRecordReader.java:136)
> > =A0 =A0 at org.apache.hadoop.mapred.MapTask$NewTrackingRecordRead= er.nextKeyValue(MapTask.java:423)
> > =A0 =A0 at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(Ma= pContext.java:67)
> > =A0 =A0 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143= )
> > =A0 =A0 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.= java:621)
> > =A0 =A0 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)=
> > =A0 =A0 at org.apache.hadoop.mapred.Child.main(Child.java:170) > > Caused by: TimedOutException()
> > =A0 =A0 at org.apache.cassandra.thrift.Cassandra$get_range_slices= _result.read(Cassandra.java:12590)
> > =A0 =A0 at org.apache.cassandra.thrift.Cassandra$Client.recv_get_= range_slices(Cassandra.java:762)
> > =A0 =A0 at org.apache.cassandra.thrift.Cassandra$Client.get_range= _slices(Cassandra.java:734)
> > =A0 =A0 at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$R= owIterator.maybeInit(ColumnFamilyRecordReader.java:243)
> > =A0 =A0 ... 11 more
> >
> > Thanks in advance,
> >
> > John
>
>
>


--bcaec51f969f2144fa04a926e045--