ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-11630) Falcon Server Seems To Have Hadoop 2.2 Paths After Upgrade
Date Wed, 03 Jun 2015 04:02:49 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-11630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570237#comment-14570237
] 

Hudson commented on AMBARI-11630:
---------------------------------

SUCCESS: Integrated in Ambari-trunk-Commit #2792 (See [https://builds.apache.org/job/Ambari-trunk-Commit/2792/])
AMBARI-11630 - Falcon Server Seems To Have Hadoop 2.2 Paths After Upgrade (jonathanhurley)
(jhurley: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=b7e1bcebf8244a63ffc594928dbb278370261c68)
* ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py
* ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
* ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
* ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py


> Falcon Server Seems To Have Hadoop 2.2 Paths After Upgrade
> ----------------------------------------------------------
>
>                 Key: AMBARI-11630
>                 URL: https://issues.apache.org/jira/browse/AMBARI-11630
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: 2.1.0
>            Reporter: Jonathan Hurley
>            Assignee: Jonathan Hurley
>            Priority: Critical
>             Fix For: 2.1.0
>
>         Attachments: AMBARI-11630.patch
>
>
> After an upgrade from HDp 2.2 to HDP 2.3, Falcon seems to have a mixture of HDP 2.2 and
2.3 paths in its process path:
> {noformat}
> [root@jhurley-hdp22-ru-6 ~]# ps aux | grep falcon
> falcon   16231  0.2  6.0 3623356 455580 ?      Sl   14:34   0:18 /usr/jdk64/jdk1.8.0_40/bin/java
-Xmx1024m -noverify -Dfalcon.embeddedmq=True -Dfalcon.emeddedmq.port=61616 -Dfalcon.log.dir=/var/log/falcon
-Dfalcon.embeddedmq.data=/hadoop/falcon/embeddedmq/data -Dfalcon.home=/usr/hdp/current/falcon-server
-Dconfig.location=/usr/hdp/current/falcon-server/conf -Dfalcon.app.type=falcon -Dfalcon.catalog.service.enabled=
-cp /usr/hdp/current/falcon-server/conf:/usr/hdp/2.2.7.0-2808/hadoop/conf:/usr/hdp/2.2.7.0-2808/hadoop/lib/*:/usr/hdp/2.2.7.0-2808/hadoop/.//*:/usr/hdp/2.2.7.0-2808/hadoop-hdfs/./:/usr/hdp/2.2.7.0-2808/hadoop-hdfs/lib/*:/usr/hdp/2.2.7.0-2808/hadoop-hdfs/.//*:/usr/hdp/2.2.7.0-2808/hadoop-yarn/lib/*:/usr/hdp/2.2.7.0-2808/hadoop-yarn/.//*:/usr/hdp/2.2.7.0-2808/hadoop-mapreduce/lib/*:/usr/hdp/2.2.7.0-2808/hadoop-mapreduce/.//*::/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/:/usr/hdp/2.2.7.0-2808/tez/*:/usr/hdp/2.2.7.0-2808/tez/lib/*:/etc/tez/conf:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/classes:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/*:/usr/hdp/current/falcon-server/libext/*
org.apache.falcon.Main -app /usr/hdp/current/falcon-server/server/webapp/falcon -port 15000
> root     28434  0.0  0.0 103252   860 pts/5    S+   16:44   0:00 grep falcon
> [root@jhurley-hdp22-ru-6 ~]# ll /usr/hdp/current | grep falcon
> lrwxrwxrwx. 1 root root 28 May 29 03:30 falcon-client -> /usr/hdp/2.2.7.0-2808/falcon
> lrwxrwxrwx. 1 root root 28 May 29 14:34 falcon-server -> /usr/hdp/2.3.0.0-2162/falcon
> {noformat}
> A restart of Falcon post-upgrade corrects these problems, so it seems like it's an issue
only during upgrade.
> Also, {{falcon-client}} is not upgraded as part of the upgrade process. This may be covered
by the {{hdp-select all}} work capture in another Jira, but falcon-server may still need the
client updated ahead of that.
> The root cause of this is the fact that Ambari is not setting {{HADOOP_HOME}} when invoking
Falcon. Under most circumstances, this is OK since Falcon's {{falcon-config.sh}} script will
then try to figure it out with {{which hadoop}}:
> {code:title=falcon-config.sh HDP 2.2}
>     HADOOPDIR=`which hadoop`
>     if [ "$HADOOPDIR" != "" ]; then
>       echo "Hadoop is installed, adding hadoop classpath to falcon classpath"
>       FALCONCPPATH="${FALCONCPPATH}:`hadoop classpath`"
>     elif [ "$HADOOP_HOME" != "" ]; then
>       echo "Hadoop home is set, adding libraries from '${HADOOP_HOME}/bin/hadoop classpath'
into falcon classpath"
>       FALCONCPPATH="${FALCONCPPATH}:`${HADOOP_HOME}/bin/hadoop classpath`"
>     else
>       echo "Could not find installed hadoop and HADOOP_HOME is not set."
>       echo "Using the default jars bundled in ${BASEDIR}/hadooplibs/"
>       FALCONCPPATH="${FALCONCPPATH}:${BASEDIR}/hadooplibs/*"
>     fi
> {code}
> The problem in HDP 2.2 is that it will always find the {{HADOOPDIR}} first and never
pickup {{HADOOP_HOME}}. In an upgraded environment, it will always use the 2.2 libs. 
> This appears to heave been fixed in HDP 2.3:
> {code:title=falcon-config.sh HDP 2.3}
>     FALCONCPPATH="$FALCON_CONF" 
>     HADOOPDIR=`which hadoop`
>     if [ "$HADOOP_HOME" != "" ]; then
>       echo "Hadoop home is set, adding libraries from '${HADOOP_HOME}/bin/hadoop classpath'
into falcon classpath"
>       FALCONCPPATH="${FALCONCPPATH}:`${HADOOP_HOME}/bin/hadoop classpath`"
>     elif [ "$HADOOPDIR" != "" ]; then
>       echo "Hadoop is installed, adding hadoop classpath to falcon classpath"
>       FALCONCPPATH="${FALCONCPPATH}:`hadoop classpath`"
>     else
>       echo "Could not find installed hadoop and HADOOP_HOME is not set."
>       echo "Using the default jars bundled in ${BASEDIR}/hadooplibs/"
>       FALCONCPPATH="${FALCONCPPATH}:${BASEDIR}/hadooplibs/*"
>     fi
> {code}
> So we just need to ensure we pass in {{HADOOP_HOME}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message