Mailing-List: contact crunch-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of josh.wills@gmail.com
 designates 209.85.217.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPP31SxcL55TzDbc3_DutDTmz056KOPTZJiwPgVEPAYbBZSZZg@mail.gmail.com>
References: 
 <CAPP31SxcL55TzDbc3_DutDTmz056KOPTZJiwPgVEPAYbBZSZZg@mail.gmail.com>
From: Josh Wills <josh.wills@gmail.com>
Date: Thu, 14 Feb 2013 06:18:05 -0800
Message-ID: 
 <CANb5z2+=0C0qzdrULNmstCYX9hsAAycHd2S=vNMPjHS58ToJ2g@mail.gmail.com>
Subject: Re: [Scrunch] implicit PTypeH value on parallelDo method
To: "crunch-user@incubator.apache.org" <crunch-user@incubator.apache.org>
Content-Type: multipart/alternative; boundary=f46d0401f97f99953f04d5afeede

--f46d0401f97f99953f04d5afeede
Content-Type: text/plain; charset=ISO-8859-1

IIRC, I couldn't put implicits on parallelDo b/c I couldn't figure out how
to make it distinguish between when it should return a PCollection vs. a
PTable. If you look at the code for (say) the map function in
PCollection.scala, you see that it looks like:

  def map[T, To](f: S => T)(implicit pt: PTypeH[T], b:
CanParallelTransform[T, To]): To = {
    b(this, mapFn(f), pt.get(getTypeFamily()))
  }

where the CanParallelTransform implicit type figures out whether or not it
should return a PCollection or a PTable based on the return type of the map
fn, which is, to be sure, really frickin' cool. If you're interested in
trying it out w/the parallelDo in PCollectionLike, I'd be curious to know
if it works, either in the current version or a later version of Scala
(2.9.2/2.10).

I'll be spending more time on Scrunch in our next release (0.6.0
timeframe), and I'm always glad to have folks who want to improve it around
to help out.

J


On Thu, Feb 14, 2013 at 2:29 AM, Lionel Herbet <lionel.herbet@gmail.com>wrote:

> Hi,
>
> I'm currently testing scrunch after few MR Job implementation on crunch.
> I was wondering if it would be possible to have implicit PTypeH parameter
> in PcollectionLike scala trait methods ?
> PCollection scala class already has implicit PtypeH parameter for map
> method.
>
> For example in PcollectionLike scala Trait we could add the method
>
> def parallelDo[T](fn: DoFn[S, T])(implicit pt: PTypeH[T]) = {
>         new PCollection[T](native.parallelDo(fn, pt.get(getTypeFamily())))
>     }
>
> This method is equivalent to the existing one but with an implicit value
> for PTypeH depending on [T] class type.
>
> def parallelDo[T](fn: DoFn[S, T], ptype: PType[T]) = {
>     new PCollection[T](native.parallelDo(fn, ptype))
>   }
>
> What do you think ?
>
> Lionel
>
>

--f46d0401f97f99953f04d5afeede
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">IIRC, I couldn&#39;t put implicits on parallelDo b/c I cou=
ldn&#39;t figure out how to make it distinguish between when it should retu=
rn a PCollection vs. a PTable. If you look at the code for (say) the map fu=
nction in PCollection.scala, you see that it looks like:<div>

<br></div><div><div>=A0 def map[T, To](f: S =3D&gt; T)(implicit pt: PTypeH[=
T], b: CanParallelTransform[T, To]): To =3D {</div><div>=A0 =A0 b(this, map=
Fn(f), pt.get(getTypeFamily()))</div><div>=A0 }</div></div><div><br></div><=
div>where the CanParallelTransform implicit type figures out whether or not=
 it should return a PCollection or a PTable based on the return type of the=
 map fn, which is, to be sure, really frickin&#39; cool. If you&#39;re inte=
rested in trying it out w/the parallelDo in PCollectionLike, I&#39;d be cur=
ious to know if it works, either in the current version or a later version =
of Scala (2.9.2/2.10).</div>

<div><br></div><div>I&#39;ll be spending more time on Scrunch in our next r=
elease (0.6.0 timeframe), and I&#39;m always glad to have folks who want to=
 improve it around to help out.</div><div><br></div><div>J</div></div>
<div class=3D"gmail_extra">
<br><br><div class=3D"gmail_quote">On Thu, Feb 14, 2013 at 2:29 AM, Lionel =
Herbet <span dir=3D"ltr">&lt;<a href=3D"mailto:lionel.herbet@gmail.com" tar=
get=3D"_blank">lionel.herbet@gmail.com</a>&gt;</span> wrote:<br><blockquote=
 class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex">

<div dir=3D"ltr"><pre>Hi,

I&#39;m currently testing scrunch after few MR Job implementation on crunch=
.
I was wondering if it would be possible to have implicit PTypeH parameter
in PcollectionLike scala trait methods ?
PCollection scala class already has implicit PtypeH parameter for map
method.

For example in PcollectionLike scala Trait we could add the method

def parallelDo[T](fn: DoFn[S, T])(implicit pt: PTypeH[T]) =3D {
        new PCollection[T](native.parallelDo(fn, pt.get(getTypeFamily())))
    }

This method is equivalent to the existing one but with an implicit value
for PTypeH depending on [T] class type.

def parallelDo[T](fn: DoFn[S, T], ptype: PType[T]) =3D {
    new PCollection[T](native.parallelDo(fn, ptype))
  }

What do you think ?

Lionel</pre><span name=3D"vitendonodeRoot"><span name=3D"vitendonodePhone">=
<span name=3D"vitendonodeText"></span></span><span title=3D"Add 0679258744 =
in Contact" name=3D"vitendonodeContact"></span><span title=3D"Call 06792587=
44" name=3D"vitendonodeCall"></span><span title=3D"Send SMS to 0679258744" =
name=3D"vitendonodeSMS"></span></span>
</div>
</blockquote></div><br></div>

--f46d0401f97f99953f04d5afeede--