Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0DE959E32 for ; Tue, 6 Mar 2012 08:44:59 +0000 (UTC) Received: (qmail 15135 invoked by uid 500); 6 Mar 2012 08:44:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 15106 invoked by uid 500); 6 Mar 2012 08:44:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 15091 invoked by uid 99); 6 Mar 2012 08:44:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Mar 2012 08:44:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of flefilla@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Mar 2012 08:44:49 +0000 Received: by werb10 with SMTP id b10so3785358wer.31 for ; Tue, 06 Mar 2012 00:44:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=KgQ3oIJd8f+GdXm3Xbmv64knyn9qjxvfoZSUsh6Dj74=; b=iFr/R68FLAamdif45cDQn3re0+HqycpUmbIqRdFio09CNDFTIITFaaxu41JzcU2kmq UG4wVu8oqUzWUT71c/1/Qvsf2NKI2u1TdqiFaaK5GvoU5oX6RgGfCMqrpDSxSpMum6j0 ymvvjgdeohKa5n6xmAHvfu68A96zgZ9SUCenSOHcJXk9SwibXMWaKbs26XjcjSlZgytc NGQxJdE/VRKMymXUYgHZtWIX1FhDeRrV74bfEmCYml2snoz88NdfEVfOUykxfRUuhp83 M2vliZR2HtiR8iIB0d3sI0ICqIkvmU1JrSJ3sAd634qqYeP4nloHbIIymAnxi2K2r2nQ pwNA== MIME-Version: 1.0 Received: by 10.180.78.130 with SMTP id b2mr17174232wix.1.1331023469647; Tue, 06 Mar 2012 00:44:29 -0800 (PST) Received: by 10.227.38.5 with HTTP; Tue, 6 Mar 2012 00:44:29 -0800 (PST) In-Reply-To: References: <76B06293-E79F-44A6-8490-788207BE26C8@gmail.com> <2C3D6FA9-3FC3-42B7-81D4-040EEB795E6C@gmail.com> <5A48F325-00B0-4783-9EE1-5DA3F7A146FB@thelastpickle.com> Date: Tue, 6 Mar 2012 09:44:29 +0100 Message-ID: Subject: Re: newer Cassandra + Hadoop = TimedOutException() From: =?ISO-8859-1?Q?Florent_Lefill=E2tre?= To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d043438ac1898f304ba8f0d20 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043438ac1898f304ba8f0d20 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, I had the same problem on hadoop 0.20.2 and cassandra 1.0.5. In my case the split of token range failed. I have comment line 'rpc_address: 0.0.0.0' in cassandra.yaml. May be see if you have not configuration changes between 0.8.7 and 0.8.10 Le 6 mars 2012 09:32, Patrik Modesto a =E9crit : > Hi, > > I was recently trying Hadoop job + cassandra-all 0.8.10 again and the > Timeouts I get are not because of the Cassandra can't handle the > requests. I've noticed there are several tasks that show proggess of > several thousands percents. Seems like they are looping their range of > keys. I've run the job with debug enabled and the ranges look ok, see > http://pastebin.com/stVsFzLM > > Another difference between cassandra-all 0.8.7 and 0.8.10 is the > number of mappers the job creates: > 0.8.7: 4680 > 0.8.10: 595 > > Task Complete > task_201202281457_2027_m_000041 9076.81% > task_201202281457_2027_m_000073 9639.04% > task_201202281457_2027_m_000105 10538.60% > task_201202281457_2027_m_000108 9364.17% > > None of this happens with cassandra-all 0.8.7. > > Regards, > P. > > > > On Tue, Feb 28, 2012 at 12:29, Patrik Modesto > wrote: > > I'll alter these settings and will let you know. > > > > Regards, > > P. > > > > On Tue, Feb 28, 2012 at 09:23, aaron morton > wrote: > >> Have you tried lowering the batch size and increasing the time out? > Even > >> just to get it to work. > >> > >> If you get a TimedOutException it means CL number of servers did not > respond > >> in time. > >> > >> Cheers > >> > >> ----------------- > >> Aaron Morton > >> Freelance Developer > >> @aaronmorton > >> http://www.thelastpickle.com > >> > >> On 28/02/2012, at 8:18 PM, Patrik Modesto wrote: > >> > >> Hi aaron, > >> > >> this is our current settings: > >> > >> > >> cassandra.range.batch.size > >> 1024 > >> > >> > >> > >> cassandra.input.split.size > >> 16384 > >> > >> > >> rpc_timeout_in_ms: 30000 > >> > >> Regards, > >> P. > >> > >> On Mon, Feb 27, 2012 at 21:54, aaron morton > wrote: > >> > >> What settings do you have for cassandra.range.batch.size > >> > >> and rpc_timeout_in_ms ? Have you tried reducing the first and/or > increasing > >> > >> the second ? > >> > >> > >> Cheers > >> > >> > >> ----------------- > >> > >> Aaron Morton > >> > >> Freelance Developer > >> > >> @aaronmorton > >> > >> http://www.thelastpickle.com > >> > >> > >> On 27/02/2012, at 8:02 PM, Patrik Modesto wrote: > >> > >> > >> On Sun, Feb 26, 2012 at 04:25, Edward Capriolo > >> > >> wrote: > >> > >> > >> Did you see the notes here? > >> > >> > >> > >> I'm not sure what do you mean by the notes? > >> > >> > >> I'm using the mapred.* settings suggested there: > >> > >> > >> > >> > >> mapred.max.tracker.failures > >> > >> 20 > >> > >> > >> > >> > >> > >> mapred.map.max.attempts > >> > >> 20 > >> > >> > >> > >> > >> > >> mapred.reduce.max.attempts > >> > >> 20 > >> > >> > >> > >> > >> But I still see the timeouts that I haven't with cassandra-all 0.8.7. > >> > >> > >> P. > >> > >> > >> http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting > >> > >> > >> > >> > --f46d043438ac1898f304ba8f0d20 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, I had the same problem on hadoop 0.20.2 and cassandra 1.0.5.
In my c= ase the split of token range failed.
I have comment line 'rpc_addres= s: 0.0.0.0' in cassandra.yaml.
May be see if you have not configurat= ion changes between 0.8.7 and 0.8.10


Le 6 mars 2012 09:32, Patrik Modesto <patrik.modes= to@gmail.com> a =E9crit :
Hi,

I was recently trying Hadoop job + cassandra-all 0.8.10 again and the
Timeouts I get are not because of the Cassandra can't handle the
requests. I've noticed there are several tasks that show proggess of several thousands percents. Seems like they are looping their range of
keys. I've run the job with debug enabled and the ranges look ok, see http://pastebin.= com/stVsFzLM

Another difference between cassandra-all 0.8.7 and 0.8.10 is the
number of mappers the job creates:
0.8.7: 4680
0.8.10: 595

Task =A0 =A0 =A0 Complete
task_201202281457_2027_m_000041 9076.81%
task_201202281457_2027_m_000073 9639.04%
task_201202281457_2027_m_000105 10538.60%
task_201202281457_2027_m_000108 9364.17%

None of this happens with cassandra-all 0.8.7.

Regards,
P.



On Tue, Feb 28, 2012 at 12:29, Patrik Modesto <patrik.modesto@gmail.com> wrote:
> I'll alter these settings and will let you know.
>
> Regards,
> P.
>
> On Tue, Feb 28, 2012 at 09:23, aaron morton <aaron@thelastpickle.com> wrote:
>> Have you tried lowering the =A0batch size and increasing the time = out? Even
>> just to get it to work.
>>
>> If you get a TimedOutException it means CL number of servers did n= ot respond
>> in time.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://= www.thelastpickle.com
>>
>> On 28/02/2012, at 8:18 PM, Patrik Modesto wrote:
>>
>> Hi aaron,
>>
>> this is our current settings:
>>
>> =A0=A0=A0=A0=A0<property>
>> =A0=A0=A0=A0=A0=A0=A0=A0=A0<name>cassandra.range.batch.size&= lt;/name>
>> =A0=A0=A0=A0=A0=A0=A0=A0=A0<value>1024</value>
>> =A0=A0=A0=A0=A0</property>
>>
>> =A0=A0=A0=A0=A0<property>
>> =A0=A0=A0=A0=A0=A0=A0=A0=A0<name>cassandra.input.split.size&= lt;/name>
>> =A0=A0=A0=A0=A0=A0=A0=A0=A0<value>16384</value>
>> =A0=A0=A0=A0=A0</property>
>>
>> rpc_timeout_in_ms: 30000
>>
>> Regards,
>> P.
>>
>> On Mon, Feb 27, 2012 at 21:54, aaron morton <aaron@thelastpickle.com> wrote:
>>
>> What settings do you have for=A0cassandra.range.batch.size
>>
>> and=A0rpc_timeout_in_ms =A0? Have you tried reducing the first and= /or increasing
>>
>> the second ?
>>
>>
>> Cheers
>>
>>
>> -----------------
>>
>> Aaron Morton
>>
>> Freelance Developer
>>
>> @aaronmorton
>>
>> http://= www.thelastpickle.com
>>
>>
>> On 27/02/2012, at 8:02 PM, Patrik Modesto wrote:
>>
>>
>> On Sun, Feb 26, 2012 at 04:25, Edward Capriolo <edlinuxguru@gmail.com>
>>
>> wrote:
>>
>>
>> Did you see the notes here?
>>
>>
>>
>> I'm not sure what do you mean by the notes?
>>
>>
>> I'm using the mapred.* settings suggested there:
>>
>>
>> =A0=A0=A0=A0<property>
>>
>> =A0=A0=A0=A0=A0=A0=A0=A0<name>mapred.max.tracker.failures<= ;/name>
>>
>> =A0=A0=A0=A0=A0=A0=A0=A0<value>20</value>
>>
>> =A0=A0=A0=A0</property>
>>
>> =A0=A0=A0=A0<property>
>>
>> =A0=A0=A0=A0=A0=A0=A0=A0<name>mapred.map.max.attempts</na= me>
>>
>> =A0=A0=A0=A0=A0=A0=A0=A0<value>20</value>
>>
>> =A0=A0=A0=A0</property>
>>
>> =A0=A0=A0=A0<property>
>>
>> =A0=A0=A0=A0=A0=A0=A0=A0<name>mapred.reduce.max.attempts<= /name>
>>
>> =A0=A0=A0=A0=A0=A0=A0=A0<value>20</value>
>>
>> =A0=A0=A0=A0</property>
>>
>>
>> But I still see the timeouts that I haven't with cassandra-all= 0.8.7.
>>
>>
>> P.
>>
>>
>> http://wiki.apache.org/cassandra/HadoopSupport#T= roubleshooting
>>
>>
>>
>>

--f46d043438ac1898f304ba8f0d20--