incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul <rsha...@xebia.com>
Subject Exception due to same iterator returned back by PGroupedTableType
Date Wed, 27 Jun 2012 11:41:00 GMT
Hi,

I am trying to create multiple iterators in a DoFn process method.

/ public void process(Pair<Integer, Iterable<TupleN>> input,
         Emitter<Pair<String, Integer>> emitter) {}

/Eve/ry /time I ask a iterator it gives back the same one and thus I 
could not not traverse the list again and again as I am hitting the 
following stack trace .
/
1575 [Thread-23] WARN  org.apache.hadoop.mapred.LocalJobRunner  - 
job_local_0001
java.util.NoSuchElementException: iterate past last value
at 
org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceContext.java:159)
at 
com.cloudera.crunch.types.PGroupedTableType$PTypeIterable$1.next(PGroupedTableType.java:56)
at 
com.mylearning.crunch.TuplesTest$ScoreCalculator.process(TuplesTest.java:52)
at 
com.mylearning.crunch.TuplesTest$ScoreCalculator.process(TuplesTest.java:1)
at com.cloudera.crunch.impl.mr.run.RTNode.process(RTNode.java:85)
at 
com.cloudera.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:39)
.............../


What I see that /PGroupedTableType /returns back the same iterator 
available to it via ReducerContext and this is the same reason that I 
get to see the exception.
Shouldn't this give a back a new iterator everytime I ask it ?
If not then what should be the way of doing the same ? (I can iterate 
first adding elements to a collection but then I donno if that should be 
the way of going forward)
I have attached my test case for your reference.

regards,
Rahul

Mime
View raw message