ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumit Mohanty" <smoha...@hortonworks.com>
Subject Re: Review Request 42340: When Spark is added later to a cluster with YARN deployed then yarn-site is not modified
Date Fri, 15 Jan 2016 23:28:21 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42340/#review114793
-----------------------------------------------------------

Ship it!


Ship It!

- Sumit Mohanty


On Jan. 15, 2016, 10:43 p.m., Swapan Shridhar wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42340/
> -----------------------------------------------------------
> 
> (Updated Jan. 15, 2016, 10:43 p.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-14679
>     https://issues.apache.org/jira/browse/AMBARI-14679
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When Spark is added later to a cluster with YARN deployed then yarn-site is not modified.
> 
> If Spark is added to an existing cluster with YARN using HDP 2.4, then yarn-site:yarn.nodemanager.aux-services
is not modified to include spark_shuffle.
> 
> Recommendation shown is :
> 
> 
> Recommendations: {'yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes':
'', 'yarn.nodemanager.container-executor.class': 'org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor',
'yarn.nodemanager.aux-services.spark_shuffle.class': 'org.apache.spark.network.yarn.YarnShuffleService',
'yarn.scheduler.minimum-allocation-vcores': '1', 'yarn.scheduler.maximum-allocation-vcores':
'1', 'yarn.nodemanager.resource.memory-mb': '1024', 'yarn.scheduler.minimum-allocation-mb':
'341', 'yarn.nodemanager.resource.cpu-vcores': '1', 'yarn.nodemanager.aux-services': 'mapreduce_shuffle,
'yarn.scheduler.maximum-allocation-mb': '1024', 'yarn.nodemanager.linux-container-executor.group':
'hadoop'}
> 
> But with SPARK added, it should be :
> 
> 'yarn.nodemanager.aux-services': 'mapreduce_shuffle,spark_shuffle'
> 
> 
> Fix : 
> 
> - Added the YARN property (yarn.nodemanager.aux-services, yarn.nodemanager.aux-services.spark_shuffle.class)
dependency on property (spark.yarn.queue).
> - Fixed the 2.4/stack_advisor code to add "spark_shuffle" value only if it doesnt exist
in property "yarn.nodemanager.aux-services".
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/configuration/yarn-site.xml
PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.4/services/stack_advisor.py 3b96268 
> 
> Diff: https://reviews.apache.org/r/42340/diff/
> 
> 
> Testing
> -------
> 
> - Tested the behavior. "spark_shuffle" gets shown in recommendations. Screenshot Attached.
>    -> New cluster install, Adding SPARK Service, deleting it and readding it.
>    -> Adding SPARK service after cluster install, deleting it and readding it.
> - Build done.
> - Python UT: Success.
> 
> 
> File Attachments
> ----------------
> 
> Screen Shot 2016-01-15 at 1.36.01 PM.png
>   https://reviews.apache.org/media/uploaded/files/2016/01/15/0396ac33-6932-461a-abba-ab2138c854ec__Screen_Shot_2016-01-15_at_1.36.01_PM.png
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message