crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinal Shah <jinalshah2...@gmail.com>
Subject Illegal State Exception when doing a union
Date Tue, 25 Feb 2014 23:04:52 GMT
Hi,

I'm trying to do an union of 3 PTables but I'm getting this error
http://pastebin.com/TkMPunJu

this is where it is throwing it
https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/org/apache/crunch/impl/dist/collect/BaseInputCollection.java#L66

this is what I'm trying to do

PCollection<U> beforeWrite = someOperation();

SourceTarget<U> target = new AvroFileTarget().asSourceTaget(U);

pipeline.write(beforeWrite, target);

PCollection<U> afterWrite = pipeline.read(target);

PCollection<U> afterParallelDo = afterWrite.parallelDo(DoFn, U,
ParallelDoOptions.builder().sources(target).build());

PTable<K,U> afterSomeOperation = someOperations();

PTable<K,U> thatNeedsToBeAdded = comingFromHbase();

PTable<K,U> unionNeeded =  afterSomeOperation.union(thatNeedsToBeAdded); //
this is where it fails for some reason since it is looking for the target
which is not generated yet.


Can anyone help me in understanding why this is happening?

Thanks
Jinal

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message