ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitro Lisnichenko <dlysniche...@hortonworks.com>
Subject Re: Review Request 42610: MapReduce, Oozie service check failures after upgrade finished
Date Fri, 05 Feb 2016 17:41:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42610/
-----------------------------------------------------------

(Updated Feb. 5, 2016, 7:41 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Jonathan Hurley, Jayush Luniya,
and Nate Cole.


Bugs: AMBARI-14764
    https://issues.apache.org/jira/browse/AMBARI-14764


Repository: ambari


Description
-------

# MapReduce jobs were failing after upgrading and restarting MapReduce2 service (ClassNotFound
exception). Also Oozie job were failing because of the same reason.
# This was caused because mapreduce.application.classpath property had the older version of
the LZO codec (/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.<OLD-STACK-VERSION>.jar).
# During upgrade the path was valid because we had not switched /usr/hdp/current/hadoop symlink
to new version. After upgrade, once the symlink is updated to new stack version and after
restarting MapReduce2 this path becomes invalid.
# To fix this updated the LZO codec path to /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar
in mapreduce.application.classpath config property. Also in oozie-env property added export
HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS”

Fix:
1. During stack upgrade, update lzo codec path in mapreduce.application.classpath to
{code}
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar
{code}

Also, looks like we also need to update following property for TEZ.

{code}
<property>
<name>tez.cluster.additional.classpath.prefix</name>
<value>/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
<description></description>
</property>
{code}

2. Update oozie-env and add
{code}
export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"
{code}

3. Add upgrade pre-checks to validate if any config properties have old-stack-version hardcodings
and report these config properties to be updated.
(attached screenshot)


Diffs (updated)
-----

  ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java 14f819e

  ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java a8c0525

  ambari-server/src/main/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheck.java
PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPath.java
PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculation.java
PRE-CREATION 
  ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml
3a3395a 
  ambari-server/src/main/resources/common-services/OOZIE/4.2.0.2.3/configuration/oozie-env.xml
93742bb 
  ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml e581506

  ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/configuration/oozie-env.xml
f0621cd 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml 098682b

  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml fe9d97c

  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml b756deb

  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.2.xml 282d143 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 133efae 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.4.xml 39756d9 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml e3b8546

  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9b36ec9

  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml a0c16a4 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 660dbe2 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 2600eae

  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 02783e9 
  ambari-server/src/test/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheckTest.java
PRE-CREATION 
  ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPathTest.java
PRE-CREATION 
  ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculationTest.java
PRE-CREATION 

Diff: https://reviews.apache.org/r/42610/diff/


Testing
-------

- Manual check of 2.3->2.4 EU upgrade on live cluster
- Manual check of 2.3->2.4 EU upgrade on live cluster
- Manual verification of a pre-upgrade check


[INFO] Reactor Summary:
[INFO] 
[INFO] Ambari Views ...................................... SUCCESS [2.012s]
[INFO] Ambari Metrics Common ............................. SUCCESS [0.886s]
[INFO] Ambari Server ..................................... SUCCESS [1:01.074s]
[INFO] Ambari Agent ...................................... SUCCESS [12.373s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:19.469s
[INFO] Finished at: Thu Feb 04 20:07:24 EET 2016
[INFO] Final Memory: 72M/1397M


File Attachments
----------------

screenshot of pre-upgrade check dialog
  https://reviews.apache.org/media/uploaded/files/2016/01/21/05890946-df65-4724-bcdd-3450b159b102__scr1.png


Thanks,

Dmitro Lisnichenko


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message