crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: [Scrunch] implicit PTypeH value on parallelDo method
Date Thu, 14 Feb 2013 14:18:05 GMT
IIRC, I couldn't put implicits on parallelDo b/c I couldn't figure out how
to make it distinguish between when it should return a PCollection vs. a
PTable. If you look at the code for (say) the map function in
PCollection.scala, you see that it looks like:

  def map[T, To](f: S => T)(implicit pt: PTypeH[T], b:
CanParallelTransform[T, To]): To = {
    b(this, mapFn(f), pt.get(getTypeFamily()))
  }

where the CanParallelTransform implicit type figures out whether or not it
should return a PCollection or a PTable based on the return type of the map
fn, which is, to be sure, really frickin' cool. If you're interested in
trying it out w/the parallelDo in PCollectionLike, I'd be curious to know
if it works, either in the current version or a later version of Scala
(2.9.2/2.10).

I'll be spending more time on Scrunch in our next release (0.6.0
timeframe), and I'm always glad to have folks who want to improve it around
to help out.

J


On Thu, Feb 14, 2013 at 2:29 AM, Lionel Herbet <lionel.herbet@gmail.com>wrote:

> Hi,
>
> I'm currently testing scrunch after few MR Job implementation on crunch.
> I was wondering if it would be possible to have implicit PTypeH parameter
> in PcollectionLike scala trait methods ?
> PCollection scala class already has implicit PtypeH parameter for map
> method.
>
> For example in PcollectionLike scala Trait we could add the method
>
> def parallelDo[T](fn: DoFn[S, T])(implicit pt: PTypeH[T]) = {
>         new PCollection[T](native.parallelDo(fn, pt.get(getTypeFamily())))
>     }
>
> This method is equivalent to the existing one but with an implicit value
> for PTypeH depending on [T] class type.
>
> def parallelDo[T](fn: DoFn[S, T], ptype: PType[T]) = {
>     new PCollection[T](native.parallelDo(fn, ptype))
>   }
>
> What do you think ?
>
> Lionel
>
>

Mime
View raw message