ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmytro Grinenko <hapy.les...@gmail.com>
Subject Re: Review Request 42610: MapReduce, Oozie service check failures after upgrade finished
Date Fri, 05 Feb 2016 15:19:06 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42610/#review118015
-----------------------------------------------------------


Ship it!




Ship It!

- Dmytro Grinenko


On Feb. 5, 2016, 3:09 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42610/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2016, 3:09 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Jonathan Hurley, Jayush
Luniya, and Nate Cole.
> 
> 
> Bugs: AMBARI-14764
>     https://issues.apache.org/jira/browse/AMBARI-14764
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> # MapReduce jobs were failing after upgrading and restarting MapReduce2 service (ClassNotFound
exception). Also Oozie job were failing because of the same reason.
> # This was caused because mapreduce.application.classpath property had the older version
of the LZO codec (/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.<OLD-STACK-VERSION>.jar).
> # During upgrade the path was valid because we had not switched /usr/hdp/current/hadoop
symlink to new version. After upgrade, once the symlink is updated to new stack version and
after restarting MapReduce2 this path becomes invalid.
> # To fix this updated the LZO codec path to /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar
in mapreduce.application.classpath config property. Also in oozie-env property added export
HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS”
> 
> Fix:
> 1. During stack upgrade, update lzo codec path in mapreduce.application.classpath to
> {code}
> /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar
> {code}
> 
> Also, looks like we also need to update following property for TEZ.
> 
> {code}
> <property>
> <name>tez.cluster.additional.classpath.prefix</name>
> <value>/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
> <description></description>
> </property>
> {code}
> 
> 2. Update oozie-env and add
> {code}
> export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"
> {code}
> 
> 3. Add upgrade pre-checks to validate if any config properties have old-stack-version
hardcodings and report these config properties to be updated.
> (attached screenshot)
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java 14f819e

>   ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java a8c0525

>   ambari-server/src/main/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheck.java
PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPath.java
PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculation.java
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml
3a3395a 
>   ambari-server/src/main/resources/common-services/OOZIE/4.2.0.2.3/configuration/oozie-env.xml
93742bb 
>   ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml
e581506 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/configuration/oozie-env.xml
f0621cd 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml
098682b 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml
fe9d97c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml
b756deb 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.2.xml 282d143 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 133efae 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.4.xml 39756d9 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
e3b8546 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
9b36ec9 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml a0c16a4 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 660dbe2 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml
2600eae 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 02783e9 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheckTest.java
PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPathTest.java
PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculationTest.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/42610/diff/
> 
> 
> Testing
> -------
> 
> - Manual check of 2.3->2.4 EU upgrade on live cluster
> - Manual check of 2.3->2.4 EU upgrade on live cluster
> - Manual verification of a pre-upgrade check
> 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Views ...................................... SUCCESS [2.012s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [0.886s]
> [INFO] Ambari Server ..................................... SUCCESS [1:01.074s]
> [INFO] Ambari Agent ...................................... SUCCESS [12.373s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1:19.469s
> [INFO] Finished at: Thu Feb 04 20:07:24 EET 2016
> [INFO] Final Memory: 72M/1397M
> 
> 
> File Attachments
> ----------------
> 
> screenshot of pre-upgrade check dialog
>   https://reviews.apache.org/media/uploaded/files/2016/01/21/05890946-df65-4724-bcdd-3450b159b102__scr1.png
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message