cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bert Passek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4229) Infinite MapReduce Task while reading via ColumnFamilyInputFormat
Date Fri, 29 Jun 2012 11:13:44 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403834#comment-13403834
] 

bert Passek commented on CASSANDRA-4229:
----------------------------------------

Hello,

i spend some time for debugging and i could find the reason for the problem described above.

This is the map function for reading from cassandra:

protected void map(ByteBuffer key, SortedMap<ByteBuffer, IColumn> values, Context context);

If you are using the row-key, you must duplicate the bytebuffer, otherwise the RowIterator
in ColumnFamilyRecordReader does not finish correctly.

This is one of the exit-condition in StaticRowIterator:

startToken = partitioner.getTokenFactory().toString(partitioner.getToken(Iterables.getLast(rows).key));
if (startToken.equals(split.getEndToken()))
{
  // reached end of the split
  rows = null;
  return;
}

Without duplicating the row-key you will never run into it. This kind of behaviour was different
to former versions of Cassandra und should be clearly documented.

Best Regards

bert Passek
                
> Infinite MapReduce Task while reading via ColumnFamilyInputFormat
> -----------------------------------------------------------------
>
>                 Key: CASSANDRA-4229
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4229
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>    Affects Versions: 1.1.0
>         Environment: Debian Squeeze
>            Reporter: bert Passek
>         Attachments: screenshot.jpg
>
>
> Hi,
> we recently upgraded cassandra from version 1.0.9 to 1.1.0. After that we can not execute
any hadoop jobs which reads data from cassandra via ColumnFamilyInputFormat.
> A map task is created which is running infinitely. We are trying to read from a super
column family with more or less 1000 row keys.
> This is the output from job interface where we already have 17 million map input records
!!!
> Map input records 	17.273.127 	0 	17.273.127
> Reduce shuffle bytes 	0 	391 	391
> Spilled Records 	3.288 	0 	3.288
> Map output bytes 	639.849.351 	0 	639.849.351
> CPU time spent (ms) 	792.750 	7.600 	800.350
> Total committed heap usage (bytes) 	354.680.832 	48.955.392 	403.636.224
> Combine input records 	17.039.783 	0 	17.039.783
> SPLIT_RAW_BYTES 	212 	0 	212
> Reduce input records 	0 	0 	0
> Reduce input groups 	0 	0 	0
> Combine output records 	3.288 	0 	3.288
> Physical memory (bytes) snapshot 	510.275.584 	96.370.688 	606.646.272
> Reduce output records 	0 	0 	0
> Virtual memory (bytes) snapshot 	1.826.496.512 	934.473.728 	2.760.970.240
> Map output records 	17.273.126 	0 	17.273.126
> We must kill the job and we have to go back to version 1.0.9 because 1.1.0 is not usable
for reading from cassandra.
> Best regards 
> Bert Passek

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message