Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 30585 invoked from network); 25 Sep 2010 01:09:58 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 25 Sep 2010 01:09:58 -0000 Received: (qmail 38617 invoked by uid 500); 25 Sep 2010 01:09:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 38573 invoked by uid 500); 25 Sep 2010 01:09:55 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 38565 invoked by uid 99); 25 Sep 2010 01:09:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Sep 2010 01:09:55 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a53.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Sep 2010 01:09:47 +0000 Received: from homiemail-a53.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a53.g.dreamhost.com (Postfix) with ESMTP id 95D50138062 for ; Fri, 24 Sep 2010 18:09:24 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=PA3UJemXoP 2qMdz8g2SLmF5qIlmZKbJKTYPNEPoSOw9L0ZBXDeMXnFEfjmdkSLsN06uoFCDMC1 E6uxXT3dwGmyX87DSvgZVm8T/i4PYY5JSAm/y711MOWjk4s6aFZW7JUph+l7tR04 ectuWPgr3GGp5I2WB/nLjv2KPD3FkzOPc= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=qItFhDYNt2UA53TY ma+UlumNeFI=; b=ZbcTlGNz7/PFHE0Vm0BwczE0N3mNtvBzPPCz6hbEYRLLogTT UCJomheymf6I/fSYDj14oMt2qyeOhyvU4G+T30c9iWlPuYo4I4GCVwIYCxYmntBH CpjRgU/nNiphzcuye3vOcJpuOWoyUFRUp4P2Kjo4DB65m3pAEbJQWSLdrW0= Received: from [10.0.1.155] (121-73-157-230.cable.telstraclear.net [121.73.157.230]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a53.g.dreamhost.com (Postfix) with ESMTPSA id 94E09138069 for ; Fri, 24 Sep 2010 18:09:23 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: multipart/alternative; boundary=Apple-Mail-1--20400729 Subject: Re: Cassandra hadoop Thrift Time out Date: Sat, 25 Sep 2010 13:09:20 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1081) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-1--20400729 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 There is some information on the wiki = http://wiki.apache.org/cassandra/HadoopSupport about a resource leak = before 0.6.2 versions that can result in a TimeoutException. But you're = on 0.6.5 so should be ok.=20 I had a quick look at the Hadoop code and could not see where to change = the timeout (that would be the obvious thing to try). If you have a look = in the ConfigHelper.java though it says=20 /** * The number of rows to request with each get range slices request. * Too big and you can either get timeouts when it takes Cassandra = too * long to fetch all the data. Too small and the performance * will be eaten up by the overhead of each request. * * @param conf Job configuration you are about to run * @param batchsize Number of rows to request each time */ public static void setRangeBatchSize(Configuration conf, int = batchsize) { conf.setInt(RANGE_BATCH_SIZE_CONFIG, batchsize); } The config item name is ""cassandra.range.batch.size". Try reducing the batch size first and see if the timeouts go away. = Though it does not sound like you have a lot of data. =20 An 0.7 beta2 may be out this week. But it's still beta.=20 Hope that helps.=20 Aaron =20 On 25 Sep 2010, at 07:17, Saket Joshi wrote: > Hi Experts, > =20 > I need help on an exception integrating cassandra-hadoop. I am getting = the following exception, when running a Hadoop Map reduce job = http://pastebin.com/RktaqDnj > I am using cassandra 0.6.5 , 3 node cluster. I don=92t get any = exception when the data I am processing is very small < 5 rows and 100 = columns, but get the error with modest data is > 5 rows 500 columns. I = went though some of the forums where people have experienced the same = issue. > = http://www.listware.net/201005/cassandra-user/21897-timeout-while-running-= simple-hadoop-job.html . Is this a bug with Cassandra-hadoop classes and = is that fixed in 0.7 for sure? how stable is 0.7 beta ? In the = system.log I see a lot of =94 index has reached its threshold; = switching in a fresh Memtable=94 messages > =20 > Has Anyone faced a similar issue and solved it? Is migrating to 0.7 = the only solution? > =20 > Thanks, > Saket > =20 > Stack Trace of the Exception: > {ava.lang.RuntimeException: TimedOutException() > at = org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit= (ColumnFamilyRecordReader.java:186) > at = org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNe= xt(ColumnFamilyRecordReader.java:236) > at = org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNe= xt(ColumnFamilyRecordReader.java:104) > at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat= or.java:135) > at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1= 30) > at = org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFa= milyRecordReader.java:98) > at = org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapT= ask.java:423) > at = org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at = org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > Caused by: TimedOutException() > at = org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassand= ra.java:11094) > at = org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassand= ra.java:628) > at = org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.ja= va:602) > at = org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit= (ColumnFamilyRecordReader.java:164)} --Apple-Mail-1--20400729 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 There is some information on the wiki http://wiki.apache= .org/cassandra/HadoopSupport about a resource leak before 0.6.2 = versions that can result in a TimeoutException. But you're on 0.6.5 so = should be ok. 

I had a quick look at the Hadoop = code and could not see where to change the timeout (that would be the = obvious thing to try). If you have a look in the ConfigHelper.java = though it = says 

 /**
   =   * The number of rows to request with each get range slices = request.
     * Too big and you can either get = timeouts when it takes Cassandra too
     * = long to fetch all the data. Too small and the = performance
     * will be eaten up by the = overhead of each request.
     = *
     * @param conf      Job = configuration you are about to run
     * = @param batchsize Number of rows to request each = time
     */
   =  public static void setRangeBatchSize(Configuration conf, int = batchsize)
    {
     =    conf.setInt(RANGE_BATCH_SIZE_CONFIG, = batchsize);
    }

The = config item name is = ""cassandra.range.batch.size".

Try reducing the = batch size first and see if the timeouts go away. Though it does not = sound like you have a lot of data.  

An = 0.7 beta2 may be out this week. But it's still = beta. 

Hope that = helps. 
Aaron


 = ; 
On 25 Sep 2010, at 07:17, Saket Joshi = wrote:

 
http://www.listware.net/201005/cassandra-user/21897-timeout-while-runnin= g-simple-hadoop-job.html . Is this a bug with = Cassandra-hadoop classes and is that fixed in 0.7 for sure? how stable = is 0.7 beta ? In the system.log I see a lot of =94  index has = reached its threshold; switching in a fresh Memtable=94 = messages
 
Thanks,
 
        at = org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFa= milyRecordReader.java:98)
        at = org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at = org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)