crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chao Shi <stepi...@live.com>
Subject Sort with multiple reducers not working?
Date Tue, 30 Jul 2013 08:06:41 GMT
Hi devs,

Does any one tried sorting with multiple reducers? I seem to hit this when
trying to implement the HFile bulk loader.

You can reproduce this as follow:
1. modify SortIT to run multiple reducers
2. run SortIT#testWritableSortDesc

I got exception:
java.lang.IllegalArgumentException: Can't read partitions file
        at
org.apache.crunch.lib.sort.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:81)
        at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:677)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.io.IOException: Wrong number of partitions in keyset
        at
org.apache.crunch.lib.sort.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:77)
        ... 6 more

It seems that TotalOrderPartitioner does not receive the correct number of
reducers. Any ideas?

Thanks,
Chao

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message