ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley" <jhur...@hortonworks.com>
Subject Review Request 36001: RU: YARN daemons started by Ambari have invalid classpath entries
Date Mon, 29 Jun 2015 16:21:48 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36001/
-----------------------------------------------------------

Review request for Ambari, Alejandro Fernandez and Nate Cole.


Bugs: AMBARI-12202
    https://issues.apache.org/jira/browse/AMBARI-12202


Repository: ambari


Description
-------

ResourceManager's command-line
```
yarn      9718  3.0  3.0 1876080 468084 ?      Sl   19:56   2:07 /usr/jdk64/jdk1.7.0_67/bin/java
-Dproc_resourcemanager -Xmx1024m -Dhdp.version=2.3.0.0-2410 -Dhadoop.log.dir=/grid/0/log/hadoop/yarn
-Dyarn.log.dir=/grid/0/log/hadoop/yarn -Dhadoop.log.file=yarn-yarn-resourcemanager-ip-172-31-39-58.log
-Dyarn.log.file=yarn-yarn-resourcemanager-ip-172-31-39-58.log -Dyarn.home.dir= -Dyarn.id.str=yarn
-Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.3.0.0-2410/hadoop/lib/native/Linux-amd64-64:/grid/0/hdp/2.3.0.0-2410/hadoop/lib/native:/grid/0/hdp/2.3.0.0-2410/hadoop/lib/native/Linux-amd64-64:/grid/0/hdp/2.3.0.0-2410/hadoop/lib/native
-Dyarn.policy.file=hadoop-policy.xml -Dyarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY
-Dhadoop.log.dir=/grid/0/log/hadoop/yarn -Dyarn.log.dir=/grid/0/log/hadoop/yarn -Dhadoop.log.file=yarn-yarn-resourcemanager-ip-172-31-39-58.log
-Dyarn.log.file=yarn-yarn-resourcemanager-ip-172-31-39-58.log -Dyarn.home.dir
 =/usr/hdp/current/hadoop-yarn-resourcemanager -Dhadoop.home.dir=/grid/0/hdp/2.3.0.0-2410/hadoop
-Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.3.0.0-2410/hadoop/lib/native/Linux-amd64-64:/grid/0/hdp/2.3.0.0-2410/hadoop/lib/native:/grid/0/hdp/2.3.0.0-2410/hadoop/lib/native/Linux-amd64-64:/grid/0/hdp/2.3.0.0-2410/hadoop/lib/native
-classpath /usr/hdp/2.3.0.0-2410/hadoop/conf:/usr/hdp/2.3.0.0-2410/hadoop/conf:/usr/hdp/2.3.0.0-2410/hadoop/conf:/usr/hdp/2.3.0.0-2410/hadoop/lib/*:/usr/hdp/2.3.0.0-2410/hadoop/.//*:/usr/hdp/2.3.0.0-2410/hadoop-hdfs/./:/usr/hdp/2.3.0.0-2410/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2410/hadoop-hdfs/.//*:/usr/hdp/2.3.0.0-2410/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2410/hadoop-yarn/.//*:/usr/hdp/2.3.0.0-2410/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2410/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/tez-cli
 ent/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/:/usr/hdp/2.3.0.0-2410/tez/*:/usr/hdp/2.3.0.0-2410/tez/lib/*:/etc/tez/conf:/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/:/usr/hdp/2.3.0.0-2410/tez/*:/usr/hdp/2.3.0.0-2410/tez/lib/*:/etc/tez/conf:/usr/hdp/current/hadoop-yarn-resourcemanager/.//*:/usr/hdp/current/hadoop-yarn-resourcemanager/lib/*:/usr/hdp/2.3.0.0-2410/hadoop/conf/rm-config/log4j.properties
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
```

Invalid entries I see
 - /usr/hdp/2.3.0.0-2410/hadoop/conf,: Three (duplicate) entries for 
 - /usr/hdp/current/hadoop-mapreduce-client/*
 - /usr/hdp/current/tez-client/*: 2 entries
 - /usr/hdp/current/tez-client/lib/*: 2 entries
 - /etc/tez/conf/: 4 entries
 - /usr/hdp/current/hadoop-mapreduce-client/*
 - /usr/hdp/current/hadoop-yarn-resourcemanager/.//*
 - /usr/hdp/current/hadoop-yarn-resourcemanager/lib/*


After upgrade, the following is the observed path:
```
/usr/jdk64/jdk1.7.0_45/bin/java -Dproc_resourcemanager -Xmx1024m -Dhdp.version=2.3.0.0-2493
-Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.dir=/var/log/hadoop-yarn/yarn -Dhadoop.log.file=yarn-yarn-resourcemanager-c6402.ambari.apache.org.log
-Dyarn.log.file=yarn-yarn-resourcemanager-c6402.ambari.apache.org.log -Dyarn.home.dir= -Dyarn.id.str=yarn
-Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.3.0.0-2493/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2493/hadoop/lib/native:/usr/hdp/2.3.0.0-2493/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2493/hadoop/lib/native
-Dyarn.policy.file=hadoop-policy.xml -Dyarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY
-Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.dir=/var/log/hadoop-yarn/yarn -Dhadoop.log.file=yarn-yarn-resourcemanager-c6402.ambari.apache.org.log
-Dyarn.log.file=yarn-yarn-resourcemanager-c6402.ambari.apache.org.log -Dyarn.home.dir=/usr/hdp/current/hadoop-yarn-reso
 urcemanager -Dhadoop.home.dir=/usr/hdp/2.3.0.0-2493/hadoop -Dhadoop.root.logger=INFO,RFA
-Dyarn.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.3.0.0-2493/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2493/hadoop/lib/native:/usr/hdp/2.3.0.0-2493/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.0.0-2493/hadoop/lib/native
-classpath /usr/hdp/2.3.0.0-2493/hadoop/conf:/usr/hdp/2.3.0.0-2493/hadoop/conf:/usr/hdp/2.3.0.0-2493/hadoop/conf:/usr/hdp/2.3.0.0-2493/hadoop/lib/*:/usr/hdp/2.3.0.0-2493/hadoop/.//*:/usr/hdp/2.3.0.0-2493/hadoop-hdfs/./:/usr/hdp/2.3.0.0-2493/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2493/hadoop-hdfs/.//*:/usr/hdp/2.3.0.0-2493/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2493/hadoop-yarn/.//*:/usr/hdp/2.3.0.0-2493/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2493/hadoop-mapreduce/.//*:/usr/hdp/current/hadoop-yarn-resourcemanager/.//*:/usr/hdp/current/hadoop-yarn-resourcemanager/lib/*:/usr/hdp/2.3.0.0-2493/hadoop/conf/rm-config/log4j.properties
org.apache.hadoop.yarn.server.resourceman
 ager.ResourceManager
```


Diffs
-----

  ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 408c593

  ambari-server/src/main/java/org/apache/ambari/server/checks/AbstractCheckDescriptor.java
8cabf29 
  ambari-server/src/main/java/org/apache/ambari/server/checks/ServicesUpCheck.java 00c798b

  ambari-server/src/main/java/org/apache/ambari/server/checks/UpgradeCheckGroup.java d46bed7

  ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java 353b84d

  ambari-server/src/main/java/org/apache/ambari/server/state/stack/PrerequisiteCheck.java
49efd4d 
  ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
80badf4 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
a54c80c 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py c526d49

  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 6330110 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml
7133c58 
  ambari-server/src/test/java/org/apache/ambari/server/checks/AbstractCheckDescriptorTest.java
4834cee 

Diff: https://reviews.apache.org/r/36001/diff/


Testing
-------

Upgrade from HDP 2.2 to HDP 2.3.

mvn clean test


Thanks,

Jonathan Hurley


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message