ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 30733: AMBARI-9494 : Support Spark installation via Ambari
Date Fri, 06 Feb 2015 18:36:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30733/#review71464
-----------------------------------------------------------

Ship it!


I'm ok with this going in now, and fixing the named imports laters.


ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml
<https://reviews.apache.org/r/30733/#comment117197>

    May as well remove this property.



ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/30733/#comment117199>

    Preferable to use named imports instead of *



ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/30733/#comment117200>

    From RU perspective, this looks good. We will need hdp-select to support the "spark" component.



ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
<https://reviews.apache.org/r/30733/#comment117203>

    Remove all of the unnecessary "pass" calls



ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
<https://reviews.apache.org/r/30733/#comment117206>

    Will need to consider this when switching between versions.



ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py
<https://reviews.apache.org/r/30733/#comment117204>

    Use named imports.



ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py
<https://reviews.apache.org/r/30733/#comment117208>

    This may need to be generalized depending on the stack type, e.g., HDP.



ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py
<https://reviews.apache.org/r/30733/#comment117209>

    Should this use hadoop-client instead?


- Alejandro Fernandez


On Feb. 6, 2015, 4:38 p.m., Gautam Borad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30733/
> -----------------------------------------------------------
> 
> (Updated Feb. 6, 2015, 4:38 p.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly and Yusaku Sako.
> 
> 
> Bugs: AMBARI-9494
>     https://issues.apache.org/jira/browse/AMBARI-9494
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Add support for management for Spark using Ambari.
> Support for following components are to be addressed :
> 1] Spark Client
> 2] Spark Job History Server
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-env.xml
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-log4j-properties.xml
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-metrics-properties.xml
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml PRE-CREATION

>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/service_check.py
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_client.py
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py
PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/30733/diff/
> 
> 
> Testing
> -------
> 
> Did the following testing:
> 
> 1] Tested on RHEL6 single node cluster 
> 2] Tested on RHEL6 multi-node cluster with Job history server on one node and spark client
on other nodes
> 3] Submitted a job using $PARK_HOME/bin/spark-submit and was able to see the job on Spark
History Server
> 
> Please note the TODO in params.py, hostLevelParams/stack_version is returning 2.2.0.0,
even if the HDP version is 2.2.1.0-xxxx
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message