Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: 
 <CAJS65-XS5Px_iQ81M-Tajty09bH2CKVgFLgYCOfoB3anuqG3-w@mail.gmail.com>
References: 
 <CAJS65-XS5Px_iQ81M-Tajty09bH2CKVgFLgYCOfoB3anuqG3-w@mail.gmail.com>
From: =?UTF-8?Q?Iulian_Drago=C8=99?= <iulian.dragos@typesafe.com>
Date: Thu, 8 Oct 2015 12:17:06 +0200
Message-ID: 
 <CAD2BF0J1Ng3GTnRyxukfedAN8p0bhLuZ8+6g=nnX3BEi29E7Pw@mail.gmail.com>
Subject: Re: Is coalesce smart while merging partitions?
To: Cesar Flores <cesar7@gmail.com>
Cc: user <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a1143a66a6bc8950521952b0c

--001a1143a66a6bc8950521952b0c
Content-Type: text/plain; charset=UTF-8

It's smart. Have a look at
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala#L123

On Thu, Oct 8, 2015 at 4:00 AM, Cesar Flores <cesar7@gmail.com> wrote:

> It is my understanding that the default behavior of coalesce function when
> the user reduce the number of partitions is to only merge them without
> executing shuffle.
>
> My question is: Is this merging smart? For example does spark try to merge
> the small partitions first or the election of partitions to merge is random?
>
>
> Thanks
> --
> Cesar Flores
>


-- 

--
Iulian Dragos

------
Reactive Apps on the JVM
www.typesafe.com

--001a1143a66a6bc8950521952b0c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">It&#39;s smart. Have a look at=C2=A0<a href=3D"https://git=
hub.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/C=
oalescedRDD.scala#L123">https://github.com/apache/spark/blob/master/core/sr=
c/main/scala/org/apache/spark/rdd/CoalescedRDD.scala#L123</a><br><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Oct 8, 2015 at 4:00=
 AM, Cesar Flores <span dir=3D"ltr">&lt;<a href=3D"mailto:cesar7@gmail.com"=
 target=3D"_blank">cesar7@gmail.com</a>&gt;</span> wrote:<br><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px=
;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1e=
x"><div dir=3D"ltr">It is my understanding that the default behavior of coa=
lesce function when the user reduce the number of partitions is to only mer=
ge them without executing shuffle.<div><br></div><div>My question is: Is th=
is merging smart? For example does spark try to merge the small partitions =
first or the election of partitions to merge is random?<br clear=3D"all"><d=
iv><br></div><div><br></div><div>Thanks</div><span class=3D""><font color=
=3D"#888888">-- <br><div>Cesar Flores</div>
</font></span></div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature"><div dir=3D"ltr"><br>--<br>Iulian Dragos<br><div><br><=
/div><div>------<br>Reactive Apps on the JVM<br><a href=3D"http://www.types=
afe.com" target=3D"_blank">www.typesafe.com</a></div><div><br></div><div><d=
iv style=3D"color:rgb(80,0,80);font-family:&#39;Helvetica Neue&#39;,Arial,s=
ans-serif;font-size:13px;font-weight:bold;overflow:hidden"></div></div></di=
v></div>
</div></div>

--001a1143a66a6bc8950521952b0c--