hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 박경희 <kh1979.p...@samsung.com>
Subject RE: how to specify cores and executor to run spark jobs simultaneously
Date Wed, 14 Sep 2016 15:09:12 GMT
<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body><font
size="2">Hi Divya</font><div><font size="2" dir="auto"><br>
</font></div><div><font size="2">I think, you did try to run spark
jobs on yarn. And also I think, you would like to submit jobs to different queues on yarn
for each. And maybe you need to&nbsp;</font><span style="font-size: 10pt;">prepare
queues on yarn to run spark jobs by configuring scheduler.</span></div><div><span
style="font-size: 10pt;"><br>
</span></div><div><font size="2">Best regards,</font></div><div><font
size="2">KyeongHee</font></div><br>
<br>
<div style="font-size:10pt;">---------<b> Original Message</b> ---------<br>
<b>Sender</b> :&nbsp;Divya Gehlot &lt;divya.htconex@gmail.com&gt;<br>
<b>Date</b> :&nbsp;2016-09-14 15:08 (GMT+9)<br>
<b>Title</b> :&nbsp;how to specify cores and executor to run spark jobs simultaneously<br>
<br>
</div><div dir="ltr">Hi,<div><br>
</div><div>I am on EMR cluster and My cluster configuration is as below:</div><div>Number
of nodes including master node - 3</div><div>Memory:<span style="font-family:sans-serif;font-size:14.4px">22.50
GB</span></div><div><span style="color:rgb(85,85,85);font-family:sans-serif;font-size:14.4px;text-align:center;background-color:rgb(230,230,230)">VCores
Total :&nbsp;</span><span style="font-family:sans-serif;font-size:14.4px">16</span><br>
</div><div><span style="font-family:sans-serif;font-size:14.4px">Active
Nodes : 2</span></div><div><font face="sans-serif"><span style="font-size:14.4px">Spark
version- 1.6.1</span></font></div><div><font face="sans-serif"><span
style="font-size:14.4px"><br>
</span></font></div><div><font face="sans-serif"><span style="font-size:14.4px">Parameter
set in spark-default.conf</span></font></div><div><font face="sans-serif"><span
style="font-size:14.4px"><br>
</span></font></div><div>







<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span
class="gmail-s1">spark.executor.instances &nbsp; &nbsp; &nbsp; &nbsp; 2<br>
</span><span class="gmail-s1">spark.executor.cores &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; 8<br>
</span><span class="gmail-s1">spark.driver.memory&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; 10473M<br>
</span><span class="gmail-s1">spark.executor.memory&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; 9658M<br>
</span><span class="gmail-s1">spark.default.parallelism&nbsp; &nbsp; &nbsp;
&nbsp; 32</span></blockquote>



</div><div><span style="font-family:sans-serif;font-size:14.4px"><br>
</span></div><div><span style="font-family:sans-serif;font-size:14.4px">Would
let me know if need any other info regarding the cluster .</span></div><div><span
style="font-family:sans-serif;font-size:14.4px"><br>
</span></div><div><span style="font-family:sans-serif;font-size:14.4px">The
current configuration for spark-submit is&nbsp;</span></div><div><div><font
face="sans-serif"><span style="font-size:14.4px">--driver-memory 5G \</span></font></div><div><font
face="sans-serif"><span style="font-size:14.4px">--executor-memory 2G \</span></font></div><div><font
face="sans-serif"><span style="font-size:14.4px">--executor-cores 5 \</span></font></div><div><font
face="sans-serif"><span style="font-size:14.4px">--num-executors 10 \</span></font></div></div><div><font
face="sans-serif"><span style="font-size:14.4px"><br>
</span></font></div><div><br>
</div><div><font face="sans-serif"><span style="font-size:14.4px">Currently
&nbsp;with the above job configuration if I try to run another spark job it will be in
accepted state till the first one finishes .</span></font></div><div><font
face="sans-serif"><span style="font-size:14.4px">How do I optimize or update the
above spark-submit configurations to run some more spark jobs simultaneously&nbsp;</span></font></div><div><font
face="sans-serif"><span style="font-size:14.4px"><br>
</span></font></div><div><font face="sans-serif"><span style="font-size:14.4px">Would
really appreciate the help.</span></font></div><div><font face="sans-serif"><span
style="font-size:14.4px"><br>
</span></font></div><div><font face="sans-serif"><span style="font-size:14.4px">Thanks,</span></font></div><div><font
face="sans-serif"><span style="font-size:14.4px">Divya&nbsp;</span></font></div><div><font
face="sans-serif"><span style="font-size:14.4px"><br>
</span></font></div></div>
<table id=bannersignimg><tr><td><div id="cafe-note-contents" style="margin:10px;font-family:Arial;font-size:10pt;"><p>&nbsp;</p></div></td></tr></table><table
id=confidentialsignimg><tr><td><div id="cafe-note-contents" style="margin:10px;font-family:Arial;font-size:10pt;"><p><img
unselectable="on" data-cui-image="true" style="display: inline-block; border: 0px solid; width:
520px; height: 144px;" src="cid:cafe_image_0@s-core.co.kr"><br></p></div></td></tr></table></body></html><img
src='http://v70ext.samsung.net/mail/ext/v1/external/status/update?userid=kh1979.park&do=bWFpbElEPTIwMTYwOTE0MTUwOTEyZXBjbXMzcDMxMGQxYWVkZjk2ZGU3MDM5YTgyZjllNzg0MjdjOWU5NSZyZWNpcGllbnRBZGRyZXNzPXVzZXJAaGFkb29wLmFwYWNoZS5vcmc_'
border=0 width=0 height=0 style='display:none'>

Mime
View raw message