ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley" <jhur...@hortonworks.com>
Subject Re: Review Request 34618: Spark History Server in HDP 2.3 no longer needs tez.tar.gz, which now needs to be uploaded to HDFS during Tez Service Check
Date Sat, 23 May 2015 01:58:34 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34618/#review85049
-----------------------------------------------------------

Ship it!


Some comments, mostly on my lack of knowledge on Spark and Tez.


ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py
<https://reviews.apache.org/r/34618/#comment136518>

    Thank you!



ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py
<https://reviews.apache.org/r/34618/#comment136519>

    I just want to make sure we have the >= correct here. Tez needs to copy its own tarball
to HDFS ... but that's not restricted to 2.3+ ... it's going to do it for 2.2 as well, right?



ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml
<https://reviews.apache.org/r/34618/#comment136520>

    Does Spark need Tez removed from its metainfo? If that's what this code is doing, then
we're OK, but at first glace, it seems like the dependency is still there.
    
    If I'm wrong, then you can drop the issue.


- Jonathan Hurley


On May 22, 2015, 9:52 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34618/
> -----------------------------------------------------------
> 
> (Updated May 22, 2015, 9:52 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-11357
>     https://issues.apache.org/jira/browse/AMBARI-11357
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> In HDP 2.3, Spark no longer needs Tez as a dependency.
> This means that
> 1. Spark metainfo.xml does not need to co-host Tez-client
> 2. Spark History Server restart or RU restart does not need to copy tez tarball to HDFS.
Note, this is still needed in HDP 2.2
> 3. Tez needs to be responsible for copying its own tarball to HDFS because its service
check depends on it.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
106b8cc 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py
834e208 
>   ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py
98973ae 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 205ccc9

>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 489037c 
> 
> Diff: https://reviews.apache.org/r/34618/diff/
> 
> 
> Testing
> -------
> 
> ----------------------------------------------------------------------
> Total run:742
> Total errors:0
> Total failures:0
> OK
> 
> Copied changes files to a cluster with HDP 2.3.0.0 and HDFS, YARN, MR, ZK.
> Then installed Spark, which did not require Tez, as expected.
> Then installed Tez, whose service checked passed.
> Then ran an RU and everything passed
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message