ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Review Request 31815: Spark on tez apps fails needs tez.tar.gz copied to HDFS
Date Fri, 06 Mar 2015 23:19:08 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31815/
-----------------------------------------------------------

Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth Gunturi, and
Sid Wagle.


Bugs: AMBARI-9954
    https://issues.apache.org/jira/browse/AMBARI-9954


Repository: ambari


Description
-------

The spark on tez apps fails because tez.tar.gz needs to be copied to HDFS.
Currently, only Pig Service Check and Hive START copy it to HDFS.


Diffs
-----

  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml ce8ad7a 
  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
518352f 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py PRE-CREATION 
  ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json PRE-CREATION


Diff: https://reviews.apache.org/r/31815/diff/


Testing
-------

I added a unit test,
----------------------------------------------------------------------
Total run:610
Total errors:0
Total failures:0
OK


I installed Spark and Tez at the same time, and put the Spark History Server on host 1, and
Tez Client on host 2. The installation for Spark History Server has a dependency on Tez Client,
so it also installed it on host 1. The Spark History Server START command then copied the
tarball to /hdp/apps/${version}/tez/tez.tar.gz , and it will also copy it during a Rolling
Upgrade restart.

```
[root@c6408 ~]# sudo su hdfs -c 'hadoop fs -ls /hdp/apps/2.2.2.0-2538/tez/'
Found 1 items
-r--r--r--   3 hdfs hadoop   40656789 2015-03-06 02:58 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz

[root@c6408 ~]# su - spark
[spark@c6408 ~]$ /usr/hdp/current/spark-client/bin/spark-submit --class org.apache.spark.examples.SparkPi
--master execution-context:org.apache.spark.tez.TezJobExecutionContext /usr/hdp/current/spark-client/lib/spark-examples-1.2.1.2.2.2.0-2538-hadoop2.6.0.2.2.2.0-2538.jar
3

[spark@c6408 ~]$ echo $?
0
```


Thanks,

Alejandro Fernandez


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message