incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Rodriguez <okramma...@gmail.com>
Subject Re: Frame size exceptions occurring with ColumnFamilyInputFormat for very large rows
Date Wed, 28 Nov 2012 04:12:46 GMT
Hi,

Even when setting it to 32m in cassandra.yaml (and restarting Cassandra), the same problem
emerges -- its as if Cassandra doesn't register the update (its always locked at 16mb). And
I know that Cassandra is reading the property from cassandra.yaml because if I do -1, it complains
saying it must be a positive value.

Apologies for the back and forth --- though I have no obvious way forward for myself.

Thank you,
Marko.

http://markorodriguez.com

P.S. Is a brontobyte an order of magnitude less than a tyranobyte?

On Nov 27, 2012, at 8:26 PM, Edward Capriolo wrote:

> Thrift has to buffer the packet into memory so setting it to 1,000,000 brontobytes is
a bad idea.
> 
> On Tue, Nov 27, 2012 at 9:17 PM, Marko Rodriguez <okrammarko@gmail.com> wrote:
> Hello,
> 
> I was wondering if anyone had an answer to my previous message below. 
> 
> Seems another is having the same problem, but unfortunately with no response as well.
> 	http://mail-archives.apache.org/mod_mbox/cassandra-user/201211.mbox/%3C509A4A1F.8070506@semantico.com%3E
> 	
> Any help would be much appreciated.
> 
> Thank you,
> Marko.
> 
> http://markorodriguez.com
> 
> On Nov 9, 2012, at 3:02 PM, Marko Rodriguez wrote:
> 
>> Hello,
>> 
>> I am trying to run a Hadoop job that pulls data out of Cassandra via ColumnFamilyInputFormat.
I am getting a "frame size" exception. To remedy that, I have set both the thrift_framed_transport_size_in_mb
and thrift_max_message_length_in_mb to an "infinite" amount at 100000mb on all nodes. Moreover,
I have restarted the cluster and the cassandra.yaml files have been reloaded.
>> 
>> However, I am still getting:
>> 
>> 12/11/09 21:39:52 INFO mapred.JobClient:  map 62% reduce 0%
>> 12/11/09 21:40:09 INFO mapred.JobClient: Task Id : attempt_201211082011_0015_m_000479_2,
Status : FAILED
>> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Frame
size (30046945) larger than max length (16384000)!
>> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
>> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
>> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:324)
>> 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>> 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:189)
>> 
>> Question: Why is 16384000 bytes (I assume) !=  100000mb?
>> 
>> Next, I made this parameter true as a last hail mary attempt:
>> 	cassandra.input.widerows=true
>> ...still with no luck.
>> 
>> Does someone know what I might be missing?
>> 
>> Thank you very much for your time,
>> Marko.
>> 
>> http://markorodriguez.com
> 
> 


Mime
View raw message