cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5529) ColumnFamilyRecordReader fails for large datasets
Date Wed, 01 May 2013 05:06:16 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646379#comment-13646379
] 

Jonathan Ellis commented on CASSANDRA-5529:
-------------------------------------------

Note to self: remove our hacked TBinaryProtocol entirely in trunk.
                
> ColumnFamilyRecordReader fails for large datasets
> -------------------------------------------------
>
>                 Key: CASSANDRA-5529
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5529
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API, Hadoop
>    Affects Versions: 0.6
>            Reporter: Rob Timpe
>            Assignee: Jonathan Ellis
>             Fix For: 1.1.12, 1.2.5
>
>         Attachments: 5529-1.1.txt, 5529.txt
>
>
> When running mapreduce jobs that read directly from cassandra, the job will sometimes
fail with an exception like this:
> java.lang.RuntimeException: com.rockmelt.org.apache.thrift.TException: Message length
exceeded: 40
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:329)
> 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:109)
> 	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522)
> 	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547)
> 	at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: com.rockmelt.org.apache.thrift.TException: Message length exceeded: 40
> 	at com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
> 	at com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
> 	at org.apache.cassandra.thrift.Column.read(Column.java:528)
> 	at org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
> 	at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408)
> 	at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12422)
> 	at com.rockmelt.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> 	at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:696)
> 	at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:680)
> 	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:362)
> 	... 16 more
> In ColumnFamilyRecordReader#initialize, a TBinaryProtocol is created as follows:
> TTransport transport = ConfigHelper.getInputTransportFactory(conf).openTransport(socket,
conf);
> TBinaryProtocol binaryProtocol = new TBinaryProtocol(transport, ConfigHelper.getThriftMaxMessageLength(conf));
> client = new Cassandra.Client(binaryProtocol);
> But each time a call to cassandra is made, checkReadLength(int length) is called in TBinaryProtocol,
which includes this:
> readLength_ -= length;
> if (readLength_ < 0) {
>    throw new TException("Message length exceeded: " + length);
> }
> The result is that readLength_ is decreased each time, until it goes negative and exception
is thrown.  This will only happen if you're reading a lot of data and your split size is large
(which is maybe why people haven't noticed it earlier).  This happens regardless of whether
you use wide row support.
> I'm not sure what the right fix is.  It seems like you could either reset the length
of TBinaryProtocol after each call or just use a new TBinaryProtocol each time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message