incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Frame size exceptions occurring with ColumnFamilyInputFormat for very large rows
Date Wed, 28 Nov 2012 04:27:36 GMT
Have you tried lowering the slice size?

On Tue, Nov 27, 2012 at 11:12 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:

> Hi,
>
> Even when setting it to 32m in cassandra.yaml (and restarting Cassandra),
> the same problem emerges -- its as if Cassandra doesn't register the update
> (its always locked at 16mb). And I know that Cassandra is reading the
> property from cassandra.yaml because if I do -1, it complains saying it
> must be a positive value.
>
> Apologies for the back and forth --- though I have no obvious way forward
> for myself.
>
> Thank you,
> Marko.
>
> http://markorodriguez.com
>
> P.S. Is a brontobyte an order of magnitude less than a tyranobyte?
>
> On Nov 27, 2012, at 8:26 PM, Edward Capriolo wrote:
>
> Thrift has to buffer the packet into memory so setting it to 1,000,000
> brontobytes is a bad idea.
>
> On Tue, Nov 27, 2012 at 9:17 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:
>
>> Hello,
>>
>> I was wondering if anyone had an answer to my previous message below.
>>
>> Seems another is having the same problem, but unfortunately with no
>> response as well.
>>
>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201211.mbox/%3C509A4A1F.8070506@semantico.com%3E
>>  Any help would be much appreciated.
>>
>> Thank you,
>> Marko.
>>
>> http://markorodriguez.com
>>
>> On Nov 9, 2012, at 3:02 PM, Marko Rodriguez wrote:
>>
>> Hello,
>>
>> I am trying to run a Hadoop job that pulls data out of Cassandra via
>> ColumnFamilyInputFormat. I am getting a "frame size" exception. To remedy
>> that, I have set both the thrift_framed_transport_size_in_mb and
>> thrift_max_message_length_in_mb to an "infinite" amount at 100000mb on all
>> nodes. Moreover, I have restarted the cluster and the cassandra.yaml files
>> have been reloaded.
>>
>> However, I am still getting:
>>
>> 12/11/09 21:39:52 INFO mapred.JobClient:  map 62% reduce 0%
>> 12/11/09 21:40:09 INFO mapred.JobClient: Task Id :
>> attempt_201211082011_0015_m_000479_2, Status : FAILED
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException: Frame size (30046945)
>> larger than max length (16384000)!
>>  at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
>>  at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:324)
>> at
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>>  at
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:189)
>>
>> Question: Why is 16384000 bytes (I assume) !=  100000mb?
>>
>> Next, I made this parameter true as a last hail mary attempt:
>> cassandra.input.widerows=true
>> ...still with no luck.
>>
>> Does someone know what I might be missing?
>>
>> Thank you very much for your time,
>> Marko.
>>
>> http://markorodriguez.com
>>
>>
>>
>
>

Mime
View raw message