Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.223.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <0D139925-F4F6-4FD4-8224-C7E3894F541C@gmail.com>
References: <48334066-2C22-411D-B2F5-DCE1EBD71375@gmail.com>
	<0D139925-F4F6-4FD4-8224-C7E3894F541C@gmail.com>
Date: Tue, 27 Nov 2012 22:26:44 -0500
Message-ID: 
 <CAENxBwzsWoywEwJNBASR6mcfC5OZQ6hLcJqiVWm=xACye8V4fA@mail.gmail.com>
Subject: Re: Frame size exceptions occurring with ColumnFamilyInputFormat for
 very large rows
From: Edward Capriolo <edlinuxguru@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=14dae93410f7602e9704cf85bcee

--14dae93410f7602e9704cf85bcee
Content-Type: text/plain; charset=ISO-8859-1

Thrift has to buffer the packet into memory so setting it to 1,000,000
brontobytes is a bad idea.

On Tue, Nov 27, 2012 at 9:17 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:

> Hello,
>
> I was wondering if anyone had an answer to my previous message below.
>
> Seems another is having the same problem, but unfortunately with no
> response as well.
>
> http://mail-archives.apache.org/mod_mbox/cassandra-user/201211.mbox/%3C509A4A1F.8070506@semantico.com%3E
> Any help would be much appreciated.
>
> Thank you,
> Marko.
>
> http://markorodriguez.com
>
> On Nov 9, 2012, at 3:02 PM, Marko Rodriguez wrote:
>
> Hello,
>
> I am trying to run a Hadoop job that pulls data out of Cassandra via
> ColumnFamilyInputFormat. I am getting a "frame size" exception. To remedy
> that, I have set both the thrift_framed_transport_size_in_mb and
> thrift_max_message_length_in_mb to an "infinite" amount at 100000mb on all
> nodes. Moreover, I have restarted the cluster and the cassandra.yaml files
> have been reloaded.
>
> However, I am still getting:
>
> 12/11/09 21:39:52 INFO mapred.JobClient:  map 62% reduce 0%
> 12/11/09 21:40:09 INFO mapred.JobClient: Task Id :
> attempt_201211082011_0015_m_000479_2, Status : FAILED
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException: Frame size (30046945)
> larger than max length (16384000)!
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:324)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:189)
>
> Question: Why is 16384000 bytes (I assume) !=  100000mb?
>
> Next, I made this parameter true as a last hail mary attempt:
> cassandra.input.widerows=true
> ...still with no luck.
>
> Does someone know what I might be missing?
>
> Thank you very much for your time,
> Marko.
>
> http://markorodriguez.com
>
>
>

--14dae93410f7602e9704cf85bcee
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thrift has to buffer the packet into memory so setting it to 1,000,000 bron=
tobytes is a bad idea.<br><br><div class=3D"gmail_quote">On Tue, Nov 27, 20=
12 at 9:17 PM, Marko Rodriguez <span dir=3D"ltr">&lt;<a href=3D"mailto:okra=
mmarko@gmail.com" target=3D"_blank">okrammarko@gmail.com</a>&gt;</span> wro=
te:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0pt 0pt 0pt 0.8ex;border-=
left:1px solid rgb(204,204,204);padding-left:1ex"><div style=3D"word-wrap:b=
reak-word">Hello,<div><br></div><div>I was wondering if anyone had an answe=
r to my previous message below.=A0</div>
<div><br></div><div>Seems another is having the same problem, but unfortuna=
tely with no response as well.</div><div><span style=3D"white-space:pre-wra=
p">	</span><a href=3D"http://mail-archives.apache.org/mod_mbox/cassandra-us=
er/201211.mbox/%3C509A4A1F.8070506@semantico.com%3E" target=3D"_blank">http=
://mail-archives.apache.org/mod_mbox/cassandra-user/201211.mbox/%3C509A4A1F=
.8070506@semantico.com%3E</a></div>
<div><span style=3D"white-space:pre-wrap">	</span></div><div>Any help would=
 be much appreciated.</div><div><br></div><div>Thank you,</div><div>Marko.<=
/div><div><br><div>
<a href=3D"http://markorodriguez.com" target=3D"_blank">http://markorodrigu=
ez.com</a>
</div>
<br><div><div>On Nov 9, 2012, at 3:02 PM, Marko Rodriguez wrote:</div><br><=
blockquote type=3D"cite"><div style=3D"word-wrap:break-word">Hello,<br><br>=
I am trying to run a Hadoop job that pulls data out of Cassandra via Column=
FamilyInputFormat. I am getting a &quot;frame size&quot; exception. To reme=
dy that, I have set both the thrift_framed_transport_size_in_mb and thrift_=
max_message_length_in_mb to an &quot;infinite&quot; amount at 100000mb on a=
ll nodes. Moreover, I have restarted the cluster and the cassandra.yaml fil=
es have been reloaded.<br>
<br>However, I am still getting:<br><br>12/11/09 21:39:52 INFO mapred.JobCl=
ient: =A0map 62% reduce 0%<br>12/11/09 21:40:09 INFO mapred.JobClient: Task=
 Id : attempt_201211082011_0015_m_000479_2, Status : FAILED<br>java.lang.Ru=
ntimeException: org.apache.thrift.transport.TTransportException: Frame size=
 (30046945) larger than max length (16384000)!<br>
<span style=3D"white-space:pre-wrap">	</span>at org.apache.cassandra.hadoop=
.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordRea=
der.java:400)<br><span style=3D"white-space:pre-wrap">	</span>at org.apache=
.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(Co=
lumnFamilyRecordReader.java:406)<br>
<span style=3D"white-space:pre-wrap">	</span>at org.apache.cassandra.hadoop=
.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordR=
eader.java:324)<br><span style=3D"white-space:pre-wrap">	</span>at com.goog=
le.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:1=
43)<br>
<span style=3D"white-space:pre-wrap">	</span>at com.google.common.collect.A=
bstractIterator.hasNext(AbstractIterator.java:138)<br><span style=3D"white-=
space:pre-wrap">	</span>at org.apache.cassandra.hadoop.ColumnFamilyRecordRe=
ader.nextKeyValue(ColumnFamilyRecordReader.java:189)<br>
<br>Question: Why is 16384000 bytes (I assume) !=3D =A0100000mb?<br><br>Nex=
t, I made this parameter true as a last hail mary attempt:<br><span style=
=3D"white-space:pre-wrap">	</span>cassandra.input.widerows=3Dtrue<br>...sti=
ll with no luck.<br>
<br>Does someone know what I might be missing?<br><br>Thank you very much f=
or your time,<br>Marko.<br><br><a href=3D"http://markorodriguez.com/" targe=
t=3D"_blank">http://markorodriguez.com</a><br></div></blockquote></div><br>
</div></div></blockquote></div><br>

--14dae93410f7602e9704cf85bcee--