Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <055B6E21-C295-46D4-BE2F-5067E18DFD9B@icloud.com>
References: <CAKKt98Sb4AtHo3wnWo=jQStZQ02btSDod5i29-=eXE=StRHfPw@mail.gmail.com>
 <055B6E21-C295-46D4-BE2F-5067E18DFD9B@icloud.com>
From: Alexander Pivovarov <apivovarov@gmail.com>
Date: Wed, 8 Jun 2016 21:51:43 -0700
Message-ID: <CAKKt98TZ0yzJvk81vGhx4RaB3jo2qmsZurmswQtkWkA_32EpNg@mail.gmail.com>
Subject: Re: rdd.distinct with Partitioner
To: =?UTF-8?B?5rGq5rSL?= <tiandiwoxin@icloud.com>
Cc: dev <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a114a6b3ee31e450534d12e12
archived-at: Thu, 09 Jun 2016 04:52:16 -0000

--001a114a6b3ee31e450534d12e12
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

reduceByKey(randomPartitioner, (a, b) =3D> a + b) also gives incorrect resu=
lt

Why reduceByKey with Partitioner exists then?

On Wed, Jun 8, 2016 at 9:22 PM, =E6=B1=AA=E6=B4=8B <tiandiwoxin@icloud.com>=
 wrote:

> Hi Alexander,
>
> I think it does not guarantee to be right if an arbitrary Partitioner is
> passed in.
>
> I have created a notebook and you can check it out. (
> https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e=
239c93eaaa8714f173bcfc/7973071962862063/2110745399505739/58107563000366/lat=
est.html
> )
>
> Best regards,
>
> Yang
>
>
> =E5=9C=A8 2016=E5=B9=B46=E6=9C=889=E6=97=A5=EF=BC=8C=E4=B8=8A=E5=8D=8811:=
42=EF=BC=8CAlexander Pivovarov <apivovarov@gmail.com> =E5=86=99=E9=81=93=EF=
=BC=9A
>
> most of the RDD methods which shuffle data take Partitioner as a paramete=
r
>
> But rdd.distinct does not have such signature
>
> Should I open a PR for that?
>
> /**
>  * Return a new RDD containing the distinct elements in this RDD.
>  */
>
> def distinct(partitioner: Partitioner)(implicit ord: Ordering[T] =3D null=
): RDD[T] =3D withScope {
>   map(x =3D> (x, null)).reduceByKey(partitioner, (x, y) =3D> x).map(_._1)
> }
>
>
>

--001a114a6b3ee31e450534d12e12
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">reduceByKey(randomPartitioner, (a, b) =3D&gt; a + b) also =
gives incorrect result=C2=A0<div><br></div><div>Why reduceByKey with Partit=
ioner exists then?<br></div></div><div class=3D"gmail_extra"><br><div class=
=3D"gmail_quote">On Wed, Jun 8, 2016 at 9:22 PM, =E6=B1=AA=E6=B4=8B <span d=
ir=3D"ltr">&lt;<a href=3D"mailto:tiandiwoxin@icloud.com" target=3D"_blank">=
tiandiwoxin@icloud.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_=
quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1=
ex"><div style=3D"word-wrap:break-word">Hi Alexander,<div><br></div><div><d=
iv>I think it does not guarantee to be right if an arbitrary Partitioner is=
 passed in.</div><div><br></div><div>I have created a notebook and you can =
check it out. (<a href=3D"https://databricks-prod-cloudfront.cloud.databric=
ks.com/public/4027ec902e239c93eaaa8714f173bcfc/7973071962862063/21107453995=
05739/58107563000366/latest.html" target=3D"_blank">https://databricks-prod=
-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/79=
73071962862063/2110745399505739/58107563000366/latest.html</a>)</div><div><=
br></div><div>Best regards,</div><div><br></div><div>Yang</div><div><div cl=
ass=3D"h5"><div><br></div><div><br></div><div><blockquote type=3D"cite"><di=
v>=E5=9C=A8 2016=E5=B9=B46=E6=9C=889=E6=97=A5=EF=BC=8C=E4=B8=8A=E5=8D=8811:=
42=EF=BC=8CAlexander Pivovarov &lt;<a href=3D"mailto:apivovarov@gmail.com" =
target=3D"_blank">apivovarov@gmail.com</a>&gt; =E5=86=99=E9=81=93=EF=BC=9A<=
/div><br><div><div dir=3D"ltr">most of the RDD methods which shuffle data t=
ake Partitioner as a parameter<div><br></div><div>But rdd.distinct does not=
 have such signature</div><div><br></div><div>Should I open a PR for that?<=
/div><div><br></div><div><pre style=3D"font-family:Menlo;font-size:9pt"><pr=
e style=3D"font-family:Menlo;font-size:9pt"><span style=3D"color:rgb(128,12=
8,128);font-style:italic">/**<br></span><span style=3D"color:rgb(128,128,12=
8);font-style:italic"> * Return a new RDD containing the distinct elements =
in this RDD.<br></span><span style=3D"color:rgb(128,128,128);font-style:ita=
lic"> */</span></pre></pre><pre style=3D"font-family:Menlo;font-size:9pt"><=
span style=3D"color:rgb(0,0,128);font-weight:bold">def </span>distinct(part=
itioner: Partitioner)(<span style=3D"color:rgb(0,0,128);font-weight:bold">i=
mplicit </span>ord: <span style=3D"color:rgb(32,153,157)">Ordering</span>[T=
] =3D <span style=3D"color:rgb(0,0,128);font-weight:bold">null</span>): RDD=
[T] =3D withScope {<br>  map(x =3D&gt; (x, <span style=3D"color:rgb(0,0,128=
);font-weight:bold">null</span>)).reduceByKey(partitioner, (x, y) =3D&gt; x=
).map(_._1)<br>}</pre></div></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br=
></div>

--001a114a6b3ee31e450534d12e12--