cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Ricardo Motta Gomes (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-6436) AbstractColumnFamilyInputFormat does not use start and end tokens configured via ConfigHelper.setInputRange()
Date Tue, 03 Dec 2013 14:37:37 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Paulo Ricardo Motta Gomes updated CASSANDRA-6436:
-------------------------------------------------

    Attachment: cassandra-1.2-6436.txt

Fix patch attached.

> AbstractColumnFamilyInputFormat does not use start and end tokens configured via ConfigHelper.setInputRange()
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6436
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6436
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>            Reporter: Paulo Ricardo Motta Gomes
>              Labels: hadoop, patch
>             Fix For: 1.2.6
>
>         Attachments: cassandra-1.2-6436.txt, cassandra-1.2-6436.txt
>
>
> ConfigHelper allows to set a token input range via the setInputRange(conf, startToken,
endToken) call (ConfigHelper:254).
> We used this feature to limit a hadoop job range to a single Cassandra node's range,
or even to single row key, mostly for testing purposes. 
> This worked before the fix for CASSANDRA-5536 (https://github.com/apache/cassandra/commit/aaf18bd08af50bbaae0954d78d5e6cbb684aded9),
but after this ColumnFamilyInputFormat never uses the value of KeyRange.start_token when defining
the input splits (AbstractColumnFamilyInputFormat:142-160), but only KeyRange.start_key, which
needs an order preserving partitioner to work.
> I propose the attached fix in order to allow defining Cassandra token ranges for a given
Hadoop job even when using a non-order preserving partitioner.
> Example use of ConfigHelper.setInputRange(conf, startToken, endToken) to limit the range
to a single Cassandra Key with RandomPartitioner: 
> IPartitioner part = ConfigHelper.getInputPartitioner(job.getConfiguration());
> Token token = part.getToken(ByteBufferUtil.bytes("Cassandra Key"));
> BigInteger endToken = (BigInteger) new BigIntegerConverter().convert(BigInteger.class,
part.getTokenFactory().toString(token));
> BigInteger startToken = endToken.subtract(new BigInteger("1"));
> ConfigHelper.setInputRange(job.getConfiguration(), startToken.toString(), endToken.toString());



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message