Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ambari.apache.org
Date: Tue, 2 Jun 2015 21:25:49 +0000 (UTC)
From: "Jonathan Hurley (JIRA)" <jira@apache.org>
To: dev@ambari.apache.org
Message-ID: <JIRA.12834823.1433280311000.2361.1433280349709@Atlassian.JIRA>
In-Reply-To: <JIRA.12834823.1433280311000@Atlassian.JIRA>
References: <JIRA.12834823.1433280311000@Atlassian.JIRA>
 <JIRA.12834823.1433280311121@arcas>
Subject: [jira] [Created] (AMBARI-11630) Falcon Server Seems To Have Hadoop
 2.2 Paths After Upgrade
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Jonathan Hurley created AMBARI-11630:
----------------------------------------

             Summary: Falcon Server Seems To Have Hadoop 2.2 Paths After Up=
grade
                 Key: AMBARI-11630
                 URL: https://issues.apache.org/jira/browse/AMBARI-11630
             Project: Ambari
          Issue Type: Bug
          Components: ambari-server
    Affects Versions: 2.1.0
            Reporter: Jonathan Hurley
            Assignee: Jonathan Hurley
            Priority: Critical
             Fix For: 2.1.0


After an upgrade from HDp 2.2 to HDP 2.3, Falcon seems to have a mixture of=
 HDP 2.2 and 2.3 paths in its process path:

{noformat}
[root@jhurley-hdp22-ru-6 ~]# ps aux | grep falcon
falcon   16231  0.2  6.0 3623356 455580 ?      Sl   14:34   0:18 /usr/jdk64=
/jdk1.8.0_40/bin/java -Xmx1024m -noverify -Dfalcon.embeddedmq=3DTrue -Dfalc=
on.emeddedmq.port=3D61616 -Dfalcon.log.dir=3D/var/log/falcon -Dfalcon.embed=
dedmq.data=3D/hadoop/falcon/embeddedmq/data -Dfalcon.home=3D/usr/hdp/curren=
t/falcon-server -Dconfig.location=3D/usr/hdp/current/falcon-server/conf -Df=
alcon.app.type=3Dfalcon -Dfalcon.catalog.service.enabled=3D -cp /usr/hdp/cu=
rrent/falcon-server/conf:/usr/hdp/2.2.7.0-2808/hadoop/conf:/usr/hdp/2.2.7.0=
-2808/hadoop/lib/*:/usr/hdp/2.2.7.0-2808/hadoop/.//*:/usr/hdp/2.2.7.0-2808/=
hadoop-hdfs/./:/usr/hdp/2.2.7.0-2808/hadoop-hdfs/lib/*:/usr/hdp/2.2.7.0-280=
8/hadoop-hdfs/.//*:/usr/hdp/2.2.7.0-2808/hadoop-yarn/lib/*:/usr/hdp/2.2.7.0=
-2808/hadoop-yarn/.//*:/usr/hdp/2.2.7.0-2808/hadoop-mapreduce/lib/*:/usr/hd=
p/2.2.7.0-2808/hadoop-mapreduce/.//*::/usr/hdp/current/hadoop-mapreduce-cli=
ent/*:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/=
tez/conf/:/usr/hdp/2.2.7.0-2808/tez/*:/usr/hdp/2.2.7.0-2808/tez/lib/*:/etc/=
tez/conf:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/classe=
s:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/*:/usr/hd=
p/current/falcon-server/libext/* org.apache.falcon.Main -app /usr/hdp/curre=
nt/falcon-server/server/webapp/falcon -port 15000
root     28434  0.0  0.0 103252   860 pts/5    S+   16:44   0:00 grep falco=
n

[root@jhurley-hdp22-ru-6 ~]# ll /usr/hdp/current | grep falcon
lrwxrwxrwx. 1 root root 28 May 29 03:30 falcon-client -> /usr/hdp/2.2.7.0-2=
808/falcon
lrwxrwxrwx. 1 root root 28 May 29 14:34 falcon-server -> /usr/hdp/2.3.0.0-2=
162/falcon
{noformat}

A restart of Falcon post-upgrade corrects these problems, so it seems like =
it's an issue only during upgrade.

Also, {{falcon-client}} is not upgraded as part of the upgrade process. Thi=
s may be covered by the {{hdp-select all}} work capture in another Jira, bu=
t falcon-server may still need the client updated ahead of that.

The root cause of this is the fact that Ambari is not setting {{HADOOP_HOME=
}} when invoking Falcon. Under most circumstances, this is OK since Falcon'=
s {{falcon-config.sh}} script will then try to figure it out with {{which h=
adoop}}:

{code:title=3Dfalcon-config.sh HDP 2.2}
    HADOOPDIR=3D`which hadoop`
    if [ "$HADOOPDIR" !=3D "" ]; then
      echo "Hadoop is installed, adding hadoop classpath to falcon classpat=
h"
      FALCONCPPATH=3D"${FALCONCPPATH}:`hadoop classpath`"
    elif [ "$HADOOP_HOME" !=3D "" ]; then
      echo "Hadoop home is set, adding libraries from '${HADOOP_HOME}/bin/h=
adoop classpath' into falcon classpath"
      FALCONCPPATH=3D"${FALCONCPPATH}:`${HADOOP_HOME}/bin/hadoop classpath`=
"
    else
      echo "Could not find installed hadoop and HADOOP_HOME is not set."
      echo "Using the default jars bundled in ${BASEDIR}/hadooplibs/"
      FALCONCPPATH=3D"${FALCONCPPATH}:${BASEDIR}/hadooplibs/*"
    fi
{code}

The problem in HDP 2.2 is that it will always find the {{HADOOPDIR}} first =
and never pickup {{HADOOP_HOME}}. In an upgraded environment, it will alway=
s use the 2.2 libs.=20

This appears to heave been fixed in HDP 2.3:

{code:title=3Dfalcon-config.sh HDP 2.3}
    FALCONCPPATH=3D"$FALCON_CONF"=20
    HADOOPDIR=3D`which hadoop`
    if [ "$HADOOP_HOME" !=3D "" ]; then
      echo "Hadoop home is set, adding libraries from '${HADOOP_HOME}/bin/h=
adoop classpath' into falcon classpath"
      FALCONCPPATH=3D"${FALCONCPPATH}:`${HADOOP_HOME}/bin/hadoop classpath`=
"
    elif [ "$HADOOPDIR" !=3D "" ]; then
      echo "Hadoop is installed, adding hadoop classpath to falcon classpat=
h"
      FALCONCPPATH=3D"${FALCONCPPATH}:`hadoop classpath`"
    else
      echo "Could not find installed hadoop and HADOOP_HOME is not set."
      echo "Using the default jars bundled in ${BASEDIR}/hadooplibs/"
      FALCONCPPATH=3D"${FALCONCPPATH}:${BASEDIR}/hadooplibs/*"
    fi
{code}

So we just need to ensure we pass in {{HADOOP_HOME}}.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)