cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Liu (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9074) Hadoop Cassandra CqlInputFormat pagination - not reading all input rows
Date Tue, 31 Mar 2015 15:26:53 GMT


Alex Liu commented on CASSANDRA-9074:

yes, please test it on the latest branch.

> Hadoop Cassandra CqlInputFormat pagination - not reading all input rows
> -----------------------------------------------------------------------
>                 Key: CASSANDRA-9074
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>         Environment: Cassandra 2.0.11, Hadoop 1.0.4, Datastax java cassandra-driver-core
>            Reporter: fuggy_yama
>            Assignee: Alex Liu
>            Priority: Minor
>             Fix For: 2.0.14
> I have a 7-node Cassandra (v2.0.11) cluster and a table with 10k rows. I run a hadoop
job (datanodes reside on cassandra nodes of course) that reads data from that table and I
see that only 7k rows is read to map phase.
> I checked CqlInputFormat source code and noticed that a CQL query is build to select
node-local date and also LIMIT clause is added (1k default). So that 7k read rows can be explained:
> 7 nodes * 1k limit = 7k rows read total
> The limit can be changed using CqlConfigHelper:
> CqlConfigHelper.setInputCQLPageRowSize(job.getConfiguration(), "1000");
> Please help me with questions below: 
> Is this a desired behavior? 
> Why CqlInputFormat does not page through the rest of rows? 
> Is it a bug or should I just increase the InputCQLPageRowSize value? 
> What if I want to read all data in table and do not know the row count?
> What if the amount of rows I need to read per cassandra node is very large - in other
words how to avoid OOM when setting InputCQLPageRowSize very large to handle all data?

This message was sent by Atlassian JIRA

View raw message