incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: [Scrunch] implicit PTypeH value on parallelDo method
Date Thu, 14 Feb 2013 14:18:05 GMT
IIRC, I couldn't put implicits on parallelDo b/c I couldn't figure out how
to make it distinguish between when it should return a PCollection vs. a
PTable. If you look at the code for (say) the map function in
PCollection.scala, you see that it looks like:

  def map[T, To](f: S => T)(implicit pt: PTypeH[T], b:
CanParallelTransform[T, To]): To = {
    b(this, mapFn(f), pt.get(getTypeFamily()))

where the CanParallelTransform implicit type figures out whether or not it
should return a PCollection or a PTable based on the return type of the map
fn, which is, to be sure, really frickin' cool. If you're interested in
trying it out w/the parallelDo in PCollectionLike, I'd be curious to know
if it works, either in the current version or a later version of Scala

I'll be spending more time on Scrunch in our next release (0.6.0
timeframe), and I'm always glad to have folks who want to improve it around
to help out.


On Thu, Feb 14, 2013 at 2:29 AM, Lionel Herbet <>wrote:

> Hi,
> I'm currently testing scrunch after few MR Job implementation on crunch.
> I was wondering if it would be possible to have implicit PTypeH parameter
> in PcollectionLike scala trait methods ?
> PCollection scala class already has implicit PtypeH parameter for map
> method.
> For example in PcollectionLike scala Trait we could add the method
> def parallelDo[T](fn: DoFn[S, T])(implicit pt: PTypeH[T]) = {
>         new PCollection[T](native.parallelDo(fn, pt.get(getTypeFamily())))
>     }
> This method is equivalent to the existing one but with an implicit value
> for PTypeH depending on [T] class type.
> def parallelDo[T](fn: DoFn[S, T], ptype: PType[T]) = {
>     new PCollection[T](native.parallelDo(fn, ptype))
>   }
> What do you think ?
> Lionel

View raw message