Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAFi_UEeaOX-r=n8DnoRsjfsK0eQidOAosCvwm0_gKXVsfnyixQ@mail.gmail.com>
References: <CAFi_UEeaOX-r=n8DnoRsjfsK0eQidOAosCvwm0_gKXVsfnyixQ@mail.gmail.com>
From: Reynold Xin <rxin@databricks.com>
Date: Fri, 20 May 2016 17:59:12 -0700
Message-ID: <CAPh_B=Y040B4N2sh0i5rG8P_t_k6NPZmfVHfwh5TcqAuSa4kDA@mail.gmail.com>
Subject: Re: Quick question on spark performance
To: Yash Sharma <yash360@gmail.com>
Cc: "dev@spark.apache.org" <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a113cd17854844905334fb804
archived-at: Sat, 21 May 2016 00:59:37 -0000

--001a113cd17854844905334fb804
Content-Type: text/plain; charset=UTF-8

It's probably due to GC.

On Fri, May 20, 2016 at 5:54 PM, Yash Sharma <yash360@gmail.com> wrote:

> Hi All,
> I am here to get some expert advice on a use case I am working on.
>
> Cluster & job details below -
>
> Data - 6 Tb
> Cluster - EMR - 15 Nodes C3-8xLarge (shared by other MR apps)
>
> Parameters-
> --executor-memory 10G \
> --executor-cores 6 \
> --conf spark.dynamicAllocation.enabled=true \
> --conf spark.dynamicAllocation.initialExecutors=15 \
>
> Runtime : 3 Hrs
>
> On monitoring the metrics I notices 10G for executors is not required
> (since I don't have lot of groupings)
>
> Reducing to --executor-memory 3G, Runtime reduced to: 2 Hrs
>
> Question:
> On adding more nodes now has absolutely no effect on the runtime. Is there
> anything I can tune/change/experiment with to make the job faster.
>
> Workload: Mostly reduceBy's and scans.
>
> Would appreciate any insights and thoughts. Best Regards
>
>
>

--001a113cd17854844905334fb804
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">It&#39;s probably due to GC.</div><div class=3D"gmail_extr=
a"><br><div class=3D"gmail_quote">On Fri, May 20, 2016 at 5:54 PM, Yash Sha=
rma <span dir=3D"ltr">&lt;<a href=3D"mailto:yash360@gmail.com" target=3D"_b=
lank">yash360@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex"><div dir=3D"ltr">Hi All,<div>I am here to get some expert advice on a =
use case I am working on.=C2=A0</div><div><br></div><div>Cluster &amp; job =
details below -</div><div><br></div><div>Data - 6 Tb</div><div>Cluster - EM=
R - 15 Nodes C3-8xLarge (shared by other MR apps)</div><div><br></div><div>=
Parameters-</div><div><div>--executor-memory 10G \</div><div>--executor-cor=
es 6 \</div><div>--conf spark.dynamicAllocation.enabled=3Dtrue \<br></div><=
div>--conf spark.dynamicAllocation.initialExecutors=3D15 \<br></div></div><=
div><br></div><div>Runtime : 3 Hrs</div><div><br></div><div>On monitoring t=
he metrics I notices 10G for executors is not required (since I don&#39;t h=
ave lot of groupings)=C2=A0</div><div><br></div><div>Reducing to --executor=
-memory 3G, Runtime reduced to: 2 Hrs</div><div><br></div><div>Question:</d=
iv><div>On adding more nodes now has absolutely no effect on the runtime. I=
s there anything I can tune/change/experiment with to make the job faster.<=
/div><div><br></div><div>Workload: Mostly reduceBy&#39;s and scans.=C2=A0</=
div><div><br></div><div>Would appreciate any insights and thoughts. Best Re=
gards</div><div><br></div><div><br></div></div>
</blockquote></div><br></div>

--001a113cd17854844905334fb804--