cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <>
Subject [jira] Commented: (CASSANDRA-2184) Returning split length of 0 confuses Pig
Date Fri, 18 Feb 2011 22:22:38 GMT


Brandon Williams commented on CASSANDRA-2184:


> Returning split length of 0 confuses Pig
> ----------------------------------------
>                 Key: CASSANDRA-2184
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.7.3
> Matt Kennedy reports on the user list,
> bq. There is a new feature in Pig 0.8 that will try to reduce the number of splits used
to speed up the whole job.  Since the ColumnFamilyInputFormat lists the input size as zero,
this feature eliminates all of the splits except for one. 
> bq. The workaround is to disable this feature for jobs that use CassandraStorage by setting
-Dpig.splitCombination=false in the pig_cassandra script.
> {noformat}
> bq. However, we wanted to keep splitCombination on because it is a useful optimization
for a lot of our use cases, so I went digging for the least intrusive way to keep the split
combiner on, but also prevent it from combining splits that read from Cassandra.  My solution,
which you are welcome to critique, is to change line 65 of
such that it returns Long.MAX_VALUE instead of zero.
> I looked into actually returning the number of keys in the split but Hadoop javadoc says
"Get the size of the split, so that the input splits can be sorted by size" so since our splits
should be very very close in size this doesn't sound like it's worth doing an extra round
trip to the host servers to get super accurate numbers on.  Returning MAX_VALUE seems like
it's good enough.

This message is automatically generated by JIRA.
For more information on JIRA, see:


View raw message