crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: Pipeline throwing No Output? exception
Date Sun, 16 Mar 2014 15:34:53 GMT
+chao

Inlined.

On Sat, Mar 15, 2014 at 12:34 PM, Jinal Shah <jinalshah2007@gmail.com>wrote:

> Hi,
>
> I actually came across a particular case I'm not sure whether the behavior
> is right or not. So here is what is happening I am getting No Output
> exception throwing while trying to run my Crunch job. On further
> investigating I found that I was using Pipeline.emptyCollection(). So here
> is how my scenario looks like
>
> PCollection<V> collectionWhichCouldBeEmpty = null;
> if(path.exists){
>     collectionWhichCouldBeEmpty= pipeline.read(FromPath, PType.V);
> } else{
>     collectionWhichCouldBeEmpty = pipeline.emptyPCollection();
> }
>
> //do some operations
>
> pipeline.write(Target);
>
> pipeline.run()// this is where it is throwing the error
>
> https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/MSCRPlanner.java#L287
>
> On further debugging I found that the Vertex didn't have an input.
>
> https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/MSCRPlanner.java#L275
>
>
> So If I use the pipeline.read and it creates an Empty PCollection it works
> since it has the input source but If I create an Empty PCollection using
> the pipeline.emptyPCollection which doesn't have an input source then it
> fails
>
> Not sure if the case is missed or it has to be like this.
>

It's a good question, and I'm not sure of the answer. Added Chao to the To:
line to ask him what the intention was in this case.


> Thanks
> Jinal
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message