ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Zhang" <zjf...@apache.org>
Subject Re: Review Request 42051: [AMBARI-14561] Configuration changes of spark thrift server for Spark 1.6
Date Tue, 19 Jan 2016 01:43:08 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42051/
-----------------------------------------------------------

(Updated Jan. 19, 2016, 1:43 a.m.)


Review request for Ambari, Alejandro Fernandez, Saisai Shao, and Sumit Mohanty.


Bugs: AMBARI-14561
    https://issues.apache.org/jira/browse/AMBARI-14561


Repository: ambari


Description
-------

This is for HDP stack, in HDP stack, we'd like to do following changes for spark thrift server
- Remove advanced properties except queue name
- Use yarn-client mode
- Enable Dynamic Resource Allocation
- Enable Fair scheduling policy


Diffs (updated)
-----

  ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py c3ffc7b

  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
943664a 
  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
68240bd 
  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py
4b38572 
  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py
d4c6732 
  ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_thrift_server.py
d2145e2 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 50c8584

  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f145de1 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-fairscheduler.xml
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml 6c0e393 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml d5e4f78 

Diff: https://reviews.apache.org/r/42051/diff/


Testing
-------

This patch doesn't solve the issue completly. The weird thing is that the configuration in
HDP 2.3's spark-thrift-sparkconf.xml will be appended to HDP-2.4's spark-thrift-sparkconf.xml.
And what I want is remove some configuration in 2.3. And actually I make Spark 1.6 of HDP
2.4 extend Spark 1.2 in common-services, not sure why it still use spark-thrift-sparkconf.xml
in HDP 2.3


Thanks,

Jeff Zhang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message