cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gijs S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4229) Infinite MapReduce Task while reading via ColumnFamilyInputFormat
Date Thu, 09 Aug 2012 15:12:19 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431887#comment-13431887
] 

Gijs S commented on CASSANDRA-4229:
-----------------------------------

I ran into the same problem:

When using cassandra 1.1.2 some map reduce jobs run infinitely. 

The problem is on the line that Bert Passek refers to:
https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob;f=src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java;h=afbda7f59cea8ecca2c7e84cfb8a04e05c0264ae;hb=dec5eedd38cb64093a965ce96b22c50bc5397b11#l338

Inserting this statement after throws an exception because the startToken is -1: partitioner.getTokenFactory().validate(startToken);

This happens in a setup with a cluster of 4 cassandra nodes and a hadoop job written with
cascalog (with a custom cassandra tap for cascading).

I haven't been able to reproduce the problem with the hadoop_word_count example.
                
> Infinite MapReduce Task while reading via ColumnFamilyInputFormat
> -----------------------------------------------------------------
>
>                 Key: CASSANDRA-4229
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4229
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>    Affects Versions: 1.1.0
>         Environment: Debian Squeeze
>            Reporter: bert Passek
>         Attachments: screenshot.jpg
>
>
> Hi,
> we recently upgraded cassandra from version 1.0.9 to 1.1.0. After that we can not execute
any hadoop jobs which reads data from cassandra via ColumnFamilyInputFormat.
> A map task is created which is running infinitely. We are trying to read from a super
column family with more or less 1000 row keys.
> This is the output from job interface where we already have 17 million map input records
!!!
> Map input records 	17.273.127 	0 	17.273.127
> Reduce shuffle bytes 	0 	391 	391
> Spilled Records 	3.288 	0 	3.288
> Map output bytes 	639.849.351 	0 	639.849.351
> CPU time spent (ms) 	792.750 	7.600 	800.350
> Total committed heap usage (bytes) 	354.680.832 	48.955.392 	403.636.224
> Combine input records 	17.039.783 	0 	17.039.783
> SPLIT_RAW_BYTES 	212 	0 	212
> Reduce input records 	0 	0 	0
> Reduce input groups 	0 	0 	0
> Combine output records 	3.288 	0 	3.288
> Physical memory (bytes) snapshot 	510.275.584 	96.370.688 	606.646.272
> Reduce output records 	0 	0 	0
> Virtual memory (bytes) snapshot 	1.826.496.512 	934.473.728 	2.760.970.240
> Map output records 	17.273.126 	0 	17.273.126
> We must kill the job and we have to go back to version 1.0.9 because 1.1.0 is not usable
for reading from cassandra.
> Best regards 
> Bert Passek

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message