crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Lewi <jer...@lewi.us>
Subject Exception with AvroPathPerKeyTarget
Date Thu, 27 Mar 2014 14:26:11 GMT
Hi

I'm hitting the exception pasted below when using AvroPathPerKeyTarget.
Interestingly, my code works just fine when I run on a small dataset using
the LocalJobTracker. However, when I run on a large dataset using a hadoop
cluster I hit the exception.

Here's a link to my code.
http://goo.gl/HTAa58

Any help would be greatly appreciated.

Thanks
Jeremy

java.io.IOException: java.lang.IllegalArgumentException: Reducer output
name 'out0' cannot be parsed
at
org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.handleMultiPaths(CrunchJobHooks.java:92)
at
org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.run(CrunchJobHooks.java:79)
at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.checkRunningState(CrunchControlledJob.java:258)
at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.checkState(CrunchControlledJob.java:268)
at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.checkRunningJobs(CrunchJobControl.java:174)
at
org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:229)
at
org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:112)
at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:55)
at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:83)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IllegalArgumentException: Reducer output name 'out0'
cannot be parsed
at
org.apache.crunch.io.impl.FileTargetImpl.extractPartitionNumber(FileTargetImpl.java:194)
at
org.apache.crunch.io.impl.FileTargetImpl.getDestFile(FileTargetImpl.java:175)
at
org.apache.crunch.io.avro.AvroPathPerKeyTarget.handleOutputs(AvroPathPerKeyTarget.java:103)
at
org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.handleMultiPaths(CrunchJobHooks.java:87)

Mime
View raw message