crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinal Shah <jinalshah2...@gmail.com>
Subject Pipeline throwing No Output? exception
Date Sat, 15 Mar 2014 19:34:44 GMT
Hi,

I actually came across a particular case I'm not sure whether the behavior
is right or not. So here is what is happening I am getting No Output
exception throwing while trying to run my Crunch job. On further
investigating I found that I was using Pipeline.emptyCollection(). So here
is how my scenario looks like

PCollection<V> collectionWhichCouldBeEmpty = null;
if(path.exists){
    collectionWhichCouldBeEmpty= pipeline.read(FromPath, PType.V);
} else{
    collectionWhichCouldBeEmpty = pipeline.emptyPCollection();
}

//do some operations

pipeline.write(Target);

pipeline.run()// this is where it is throwing the error
https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/MSCRPlanner.java#L287

On further debugging I found that the Vertex didn't have an input.
https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/MSCRPlanner.java#L275


So If I use the pipeline.read and it creates an Empty PCollection it works
since it has the input source but If I create an Empty PCollection using
the pipeline.emptyPCollection which doesn't have an input source then it
fails

Not sure if the case is missed or it has to be like this.

Thanks
Jinal

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message