Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ambari.apache.org
Content-Type: multipart/alternative;
 boundary="===============4268361179275228741=="
MIME-Version: 1.0
Subject: Re: Review Request 42610: [preview] MapReduce,
 Oozie service check failures after upgrade finished
From: Dmitro Lisnichenko <dlysnichenko@hortonworks.com>
To: Jayush Luniya <jluniya@hortonworks.com>,
 Jonathan Hurley <jhurley@hortonworks.com>,
 Dmytro Grinenko <hapy.lestat@gmail.com>,
 Alejandro Fernandez <afernandez@hortonworks.com>,
 Nate Cole <ncole@hortonworks.com>
Cc: Ambari <dev@ambari.apache.org>
Date: Fri, 05 Feb 2016 14:19:02 -0000
Message-ID: <20160205141902.1669.36370@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: Dmitro Lisnichenko <noreply@reviews.apache.org>
References: <20160205141746.1669.66305@reviews.apache.org>
In-Reply-To: <20160205141746.1669.66305@reviews.apache.org>
Reply-To: Dmitro Lisnichenko <dlysnichenko@hortonworks.com>

--===============4268361179275228741==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42610/
-----------------------------------------------------------

(Updated Feb. 5, 2016, 4:18 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.


Bugs: AMBARI-14764
    https://issues.apache.org/jira/browse/AMBARI-14764


Repository: ambari


Description
-------

# MapReduce jobs were failing after upgrading and restarting MapReduce2 service (ClassNotFound exception). Also Oozie job were failing because of the same reason.
# This was caused because mapreduce.application.classpath property had the older version of the LZO codec (/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.<OLD-STACK-VERSION>.jar).
# During upgrade the path was valid because we had not switched /usr/hdp/current/hadoop symlink to new version. After upgrade, once the symlink is updated to new stack version and after restarting MapReduce2 this path becomes invalid.
# To fix this updated the LZO codec path to /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar in mapreduce.application.classpath config property. Also in oozie-env property added export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS”

Fix:
1. During stack upgrade, update lzo codec path in mapreduce.application.classpath to
{code}
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar
{code}

Also, looks like we also need to update following property for TEZ.

{code}
<property>
<name>tez.cluster.additional.classpath.prefix</name>
<value>/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
<description></description>
</property>
{code}

2. Update oozie-env and add
{code}
export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"
{code}

3. Add upgrade pre-checks to validate if any config properties have old-stack-version hardcodings and report these config properties to be updated.
(attached screenshot)


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java 14f819e 
  ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java a8c0525 
  ambari-server/src/main/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheck.java PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPath.java PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculation.java PRE-CREATION 
  ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml 3a3395a 
  ambari-server/src/main/resources/common-services/OOZIE/4.2.0.2.3/configuration/oozie-env.xml 93742bb 
  ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml e581506 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/configuration/oozie-env.xml f0621cd 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml 098682b 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml fe9d97c 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml b756deb 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.2.xml 282d143 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 133efae 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.4.xml 39756d9 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml e3b8546 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 9b36ec9 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml a0c16a4 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 660dbe2 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 2600eae 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 02783e9 
  ambari-server/src/test/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheckTest.java PRE-CREATION 
  ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPathTest.java PRE-CREATION 
  ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculationTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/42610/diff/


Testing (updated)
-------

- Manual check of 2.3->2.4 EU upgrade on live cluster
- Manual check of 2.3->2.4 EU upgrade on live cluster
- Manual verification of a pre-upgrade check


[INFO] Reactor Summary:
[INFO] 
[INFO] Ambari Views ...................................... SUCCESS [2.012s]
[INFO] Ambari Metrics Common ............................. SUCCESS [0.886s]
[INFO] Ambari Server ..................................... SUCCESS [1:01.074s]
[INFO] Ambari Agent ...................................... SUCCESS [12.373s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:19.469s
[INFO] Finished at: Thu Feb 04 20:07:24 EET 2016
[INFO] Final Memory: 72M/1397M


File Attachments
----------------

screenshot of pre-upgrade check dialog
  https://reviews.apache.org/media/uploaded/files/2016/01/21/05890946-df65-4724-bcdd-3450b159b102__scr1.png


Thanks,

Dmitro Lisnichenko


--===============4268361179275228741==--