ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 31815: Spark on tez apps fails needs tez.tar.gz copied to HDFS
Date Sat, 07 Mar 2015 00:29:58 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31815/
-----------------------------------------------------------

(Updated March 7, 2015, 12:29 a.m.)


Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth Gunturi, and
Sid Wagle.


Bugs: AMBARI-9954
    https://issues.apache.org/jira/browse/AMBARI-9954


Repository: ambari


Description
-------

The spark on tez apps fails because tez.tar.gz needs to be copied to HDFS.
Currently, only Pig Service Check and Hive START copy it to HDFS.


Diffs
-----

  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml ce8ad7a 
  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
518352f 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py PRE-CREATION 
  ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json PRE-CREATION


Diff: https://reviews.apache.org/r/31815/diff/


Testing (updated)
-------

I added a unit test,
----------------------------------------------------------------------
Total run:610
Total errors:0
Total failures:0
OK

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.420s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.032s]
[INFO] Ambari Web ........................................ SUCCESS [41.221s]
[INFO] Ambari Views ...................................... SUCCESS [2.345s]
[INFO] Ambari Admin View ................................. SUCCESS [10.019s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.446s]
[INFO] Ambari Server ..................................... SUCCESS [33:24.375s]
[INFO] Ambari Agent ...................................... SUCCESS [16.789s]
[INFO] Ambari Client ..................................... SUCCESS [0.038s]
[INFO] Ambari Python Client .............................. SUCCESS [0.257s]
[INFO] Ambari Groovy Client .............................. SUCCESS [9.922s]
[INFO] Ambari Shell ...................................... SUCCESS [0.033s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.036s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [6.632s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35:01.956s
[INFO] Finished at: Fri Mar 06 16:16:31 PST 2015
[INFO] Final Memory: 69M/565M
[INFO] ------------------------------------------------------------------------


I installed Spark and Tez at the same time, and put the Spark History Server on host 1, and
Tez Client on host 2. The installation for Spark History Server has a dependency on Tez Client,
so it also installed it on host 1. The Spark History Server START command then copied the
tarball to /hdp/apps/${version}/tez/tez.tar.gz , and it will also copy it during a Rolling
Upgrade restart.

```
[root@c6408 ~]# sudo su hdfs -c 'hadoop fs -ls /hdp/apps/2.2.2.0-2538/tez/'
Found 1 items
-r--r--r--   3 hdfs hadoop   40656789 2015-03-06 02:58 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz

[root@c6408 ~]# su - spark
[spark@c6408 ~]$ /usr/hdp/current/spark-client/bin/spark-submit --class org.apache.spark.examples.SparkPi
--master execution-context:org.apache.spark.tez.TezJobExecutionContext /usr/hdp/current/spark-client/lib/spark-examples-1.2.1.2.2.2.0-2538-hadoop2.6.0.2.2.2.0-2538.jar
3

[spark@c6408 ~]$ echo $?
0
```


Thanks,

Alejandro Fernandez


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message