incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Frame size exceptions occurring with ColumnFamilyInputFormat for very large rows
Date Wed, 28 Nov 2012 03:26:44 GMT
Thrift has to buffer the packet into memory so setting it to 1,000,000
brontobytes is a bad idea.

On Tue, Nov 27, 2012 at 9:17 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:

> Hello,
>
> I was wondering if anyone had an answer to my previous message below.
>
> Seems another is having the same problem, but unfortunately with no
> response as well.
>
> http://mail-archives.apache.org/mod_mbox/cassandra-user/201211.mbox/%3C509A4A1F.8070506@semantico.com%3E
> Any help would be much appreciated.
>
> Thank you,
> Marko.
>
> http://markorodriguez.com
>
> On Nov 9, 2012, at 3:02 PM, Marko Rodriguez wrote:
>
> Hello,
>
> I am trying to run a Hadoop job that pulls data out of Cassandra via
> ColumnFamilyInputFormat. I am getting a "frame size" exception. To remedy
> that, I have set both the thrift_framed_transport_size_in_mb and
> thrift_max_message_length_in_mb to an "infinite" amount at 100000mb on all
> nodes. Moreover, I have restarted the cluster and the cassandra.yaml files
> have been reloaded.
>
> However, I am still getting:
>
> 12/11/09 21:39:52 INFO mapred.JobClient:  map 62% reduce 0%
> 12/11/09 21:40:09 INFO mapred.JobClient: Task Id :
> attempt_201211082011_0015_m_000479_2, Status : FAILED
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException: Frame size (30046945)
> larger than max length (16384000)!
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:324)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:189)
>
> Question: Why is 16384000 bytes (I assume) !=  100000mb?
>
> Next, I made this parameter true as a last hail mary attempt:
> cassandra.input.widerows=true
> ...still with no luck.
>
> Does someone know what I might be missing?
>
> Thank you very much for your time,
> Marko.
>
> http://markorodriguez.com
>
>
>

Mime
View raw message