Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 9955 invoked from network); 18 Aug 2010 13:25:04 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Aug 2010 13:25:04 -0000 Received: (qmail 46801 invoked by uid 500); 18 Aug 2010 13:25:02 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 46768 invoked by uid 500); 18 Aug 2010 13:25:00 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 46760 invoked by uid 99); 18 Aug 2010 13:25:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 13:25:00 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of decker.christian@gmail.com designates 209.85.216.179 as permitted sender) Received: from [209.85.216.179] (HELO mail-qy0-f179.google.com) (209.85.216.179) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 13:24:55 +0000 Received: by qyk9 with SMTP id 9so575382qyk.10 for ; Wed, 18 Aug 2010 06:24:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=SczRjW92Iny2Dxia4nU7pSu6/wFPMGGNZugYl0WHYq0=; b=FV1AdBhRIfuV0I/wfRtdxyUp3rNHANnGKVLEPGUJfd/oF2oFKxYCjbEdkXSwVv7y3p wUzRNlVSM2lOofP8M5uyXg6ptRQMuy1Es+uhLt4FFHQycHRm6EcdMah+5GEMPYGr9Orm WSvL0SFkDhQQD7htjaUFbxRhIHDTIpMjEbFyc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=cvslWCzS3dwNiFTEAE+z4mFnV90BVvXYYKfDMFgonDvXVfW6aBlvxZEFhavQESgo3g vtdpJTU6oXJyqk27zbB6/K2En3/Lccs2CAUHQbueRUp/v7sFqEZ69TtSmIxXxFgEb8YH /gRwrsJWU9H9adeVh6bfzo0oJWMfZaMV1Hocs= Received: by 10.224.119.20 with SMTP id x20mr5331550qaq.249.1282137874198; Wed, 18 Aug 2010 06:24:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.110.5 with HTTP; Wed, 18 Aug 2010 06:24:12 -0700 (PDT) In-Reply-To: References: From: Christian Decker Date: Wed, 18 Aug 2010 15:24:12 +0200 Message-ID: Subject: Re: Pig + Cassandra = Connection errors To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0014852babc78b85dc048e18fcee --0014852babc78b85dc048e18fcee Content-Type: text/plain; charset=ISO-8859-1 You mean the ? Right now it's 10000 milliseconds. So that should take care of the timeouts, but what about the refused connections? On Wed, Aug 18, 2010 at 3:08 PM, Drew Dahlke wrote: > What's your cassandra timeout configured to? It's not uncommon to > raise that to 30sec if you're getting timeouts. > > On Wed, Aug 18, 2010 at 8:17 AM, Christian Decker > wrote: > > Hi all, > > I'm trying to get Pig scripts to work on data in Cassandra and right now > I > > want to simply run the example-script.pig on a different Keyspace/CF > > containing ~6'000'000 entries. I got it running but then the job aborts > > after quite some time, and when I look at the logs I see hundreds of > these: > >> > >> java.lang.RuntimeException: > >> org.apache.thrift.transport.TTransportException: > java.net.ConnectException: > >> Connection refused > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:133) > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:224) > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:101) > >> at > >> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) > >> at > >> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:95) > >> at org.apache.cassandra.hadoop.pig.CassandraStorage.getNext(Unknown > >> Source) > >> at > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:142) > >> at > >> > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423) > >> at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > >> at org.apache.hadoop.mapred.Child.main(Child.java:170) > >> Caused by: org.apache.thrift.transport.TTransportException: > >> java.net.ConnectException: Connection refused > >> at org.apache.thrift.transport.TSocket.open(TSocket.java:185) > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:129) > >> ... 13 more > >> Caused by: java.net.ConnectException: Connection refused > >> at java.net.PlainSocketImpl.socketConnect(Native Method) > >> at > >> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310) > >> at > >> > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176) > >> at > >> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163) > >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:381) > >> at java.net.Socket.connect(Socket.java:537) > >> at java.net.Socket.connect(Socket.java:487) > >> at org.apache.thrift.transport.TSocket.open(TSocket.java:180) > >> ... 14 more > > > > and > >> > >> > >> > >> java.lang.RuntimeException: TimedOutException() > >> > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:174) > >> > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:224) > >> > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:101) > >> > >> at > >> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) > >> > >> at > >> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) > >> > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:95) > >> > >> at org.apache.cassandra.hadoop.pig.CassandraStorage.getNext(Unknown > >> Source) > >> > >> at > >> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:142) > >> > >> at > >> > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423) > >> > >> at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > >> > >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > >> > >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) > >> > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > >> > >> at org.apache.hadoop.mapred.Child.main(Child.java:170) > >> > >> Caused by: TimedOutException() > >> > >> at > >> > org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:11030) > >> > >> at > >> > org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:623) > >> > >> at > >> > org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:597) > >> > >> at > >> > org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:151) > >> > >> ... 13 more > > > > I checked that the cassandra cluster is running and all my 3 nodes are up > > and working. As far as I see it the Jobtracker retries when it get those > > errors but aborts once a large portion have failed. Any idea on why the > > Cluster keeps dropping connections or timing out? > > Regards, > > Chris > > -- > > Christian Decker > > Software Architect > > http://blog.snyke.net > > > --0014852babc78b85dc048e18fcee Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You mean the <RpcTimeoutInMillis/>? Right now it's 10000 millisec= onds. So that should take care of the timeouts, but what about the refused = connections?

On Wed, Aug 18, 2010 at 3:08= PM, Drew Dahlke <drew.dahlke@bronto.com> wrote:
What's your cassandra timeout configure= d to? It's not uncommon to
raise that to 30sec if you're getting timeouts.

On Wed, Aug 18, 2010 at 8:17 AM, Christian Decker
<decker.christian@gmail.co= m> wrote:
> Hi all,
> I'm trying to get Pig scripts to work on data in Cassandra and rig= ht now I
> want to simply run the example-script.pig on a different Keyspace/CF > containing ~6'000'000 entries. I got it running but then the j= ob aborts
> after quite some time, and when I look at the logs I see hundreds of t= hese:
>>
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException: java.net.ConnectE= xception:
>> Connection refused
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.m= aybeInit(ColumnFamilyRecordReader.java:133)
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.c= omputeNext(ColumnFamilyRecordReader.java:224)
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.c= omputeNext(ColumnFamilyRecordReader.java:101)
>> at
>> com.google.common.collect.AbstractIterator.tryToComputeNext(Abstra= ctIterator.java:135)
>> at
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterato= r.java:130)
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(= ColumnFamilyRecordReader.java:95)
>> at org.apache.cassandra.hadoop.pig.CassandraStorage.getNext(Unknow= n
>> Source)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRe= cordReader.nextKeyValue(PigRecordReader.java:142)
>> at
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyVa= lue(MapTask.java:423)
>> at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.= java:67)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)=
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused
>> at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.m= aybeInit(ColumnFamilyRecordReader.java:129)
>> ... 13 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl= .java:310)
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSoc= ketImpl.java:176)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.j= ava:163)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:381)
>> at java.net.Socket.connect(Socket.java:537)
>> at java.net.Socket.connect(Socket.java:487)
>> at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
>> ... 14 more
>
> and
>>
>>
>>
>> java.lang.RuntimeException: TimedOutException()
>>
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.m= aybeInit(ColumnFamilyRecordReader.java:174)
>>
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.c= omputeNext(ColumnFamilyRecordReader.java:224)
>>
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.c= omputeNext(ColumnFamilyRecordReader.java:101)
>>
>> at
>> com.google.common.collect.AbstractIterator.tryToComputeNext(Abstra= ctIterator.java:135)
>>
>> at
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterato= r.java:130)
>>
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(= ColumnFamilyRecordReader.java:95)
>>
>> at org.apache.cassandra.hadoop.pig.CassandraStorage.getNext(Unknow= n
>> Source)
>>
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRe= cordReader.nextKeyValue(PigRecordReader.java:142)
>>
>> at
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyVa= lue(MapTask.java:423)
>>
>> at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.= java:67)
>>
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>>
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)=
>>
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>> Caused by: TimedOutException()
>>
>> at
>> org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read= (Cassandra.java:11030)
>>
>> at
>> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices= (Cassandra.java:623)
>>
>> at
>> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cass= andra.java:597)
>>
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.m= aybeInit(ColumnFamilyRecordReader.java:151)
>>
>> ... 13 more
>
> I checked that the cassandra cluster is running and all my 3 nodes are= up
> and working. As far as I see it the Jobtracker retries when it get tho= se
> errors but aborts once a large portion have failed. Any idea on why th= e
> Cluster keeps dropping connections or timing out?
> Regards,
> Chris
> --
> Christian Decker
> Software Architect
> http://blog.snyke.= net
>

--0014852babc78b85dc048e18fcee--