ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Di Li (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMBARI-21607) Hive can not write to session directory after upgrade
Date Fri, 04 Aug 2017 14:14:00 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-21607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114416#comment-16114416
] 

Di Li edited comment on AMBARI-21607 at 8/4/17 2:13 PM:
--------------------------------------------------------

Hello Eric and Attila,

_hbase_lib_ was removed in BI 4.2.5. We should not use it post migration. The following section
in hive-env from IOP 4.2 should be removed during EU.

{quote}# Set HIVE_AUX_JARS_PATH
export HIVE_AUX_JARS_PATH={{hbase_lib}}/hbase-client.jar,\
{{hbase_lib}}/hbase-common.jar,\
{{hbase_lib}}/hbase-hadoop2-compat.jar,\
{{hbase_lib}}/hbase-prefix-tree.jar,\
{{hbase_lib}}/hbase-protocol.jar,\
{{hbase_lib}}/hbase-server.jar,\
{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\
${HIVE_AUX_JARS_PATH}
{quote}

Right around the time of the IOP/HDP annoucement, Tim,  myself, and a dev from our Hive team
were looking at a fix and our thinking was the following 

{quote}<service name="HIVE">
      <component name="HIVE_SERVER">
        <changes>
          <definition xsi:type="configure" id="biginsights_4_2_5_update_hive_env">
            <type>hive-env</type>
            <replace key="content" find="{{hbase_lib}}/hbase-client.jar," replace-with=""/>
            <replace key="content" find="{{hbase_lib}}/hbase-common.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-hadoop2-compat.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-prefix-tree.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-protocol.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-server.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\"
replace-with="\"/>
          </definition>{quote}

So the end result is like the following
{quote}export HIVE_AUX_JARS_PATH=\
\
\
\
\
\
\
${HIVE_AUX_JARS_PATH}{quote}
		  
*IMPORTANT*: Please note that for a BigSQL cluster to be migrated, hive-env has the following
at the end of the file that *must be retained during EU*. -- the section only exists on a
BigSQL cluster.

{quote}# Allow Hive to read Big SQL HBase tables
if [ -d "/usr/ibmpacks/current/bigsql/bigsql/lib/java" ]; then
export HIVE_AUX_JARS_PATH=\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-io.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-hbase.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/commoncatalog.jar,\
/usr/ibmpacks/current/bigsql/hive/lib/hive-hbase-handler.jar,\
${HIVE_AUX_JARS_PATH}
fi{quote}

P.S
I have an IOP4.2.5/BigSQL cluster migrated to HDP 2.6.2 and I am able to open the Hive shell
after I switch (the migration was done before Attila's EU config fix) the engine to MR.

[hive@loch4 ~]$ hive
2017-08-04 06:40:32,485 WARN  [main] conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce
does not exist
Logging initialized using configuration in file:/etc/hive/2.6.2.0-129/0/hive-log4j.properties
hive>

Here is what its hive aux lib setting looks like - the bigsql section at the end

"""
export HIVE_AUX_JARS_PATH="{{stack_root}}/current/ext/hive"

if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
  if [ -f "${HIVE_AUX_JARS_PATH}" ] || [ -d "${HIVE_AUX_JARS_PATH}" ] ; then
    export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
  elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
    export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar
  fi
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
  export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar
fi

export METASTORE_PORT={{hive_metastore_port}}

{% if sqla_db_used or lib_dir_available %}
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:{{jdbc_libs_dir}}"
export JAVA_LIBRARY_PATH="$JAVA_LIBRARY_PATH:{{jdbc_libs_dir}}"
{% endif %}

# Allow Hive to read Big SQL HBase tables
if [ -d "/usr/ibmpacks/current/bigsql/bigsql/lib/java" ]; then
export HIVE_AUX_JARS_PATH=\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-io.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-hbase.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/commoncatalog.jar,\
/usr/ibmpacks/current/bigsql/hive/lib/hive-hbase-handler.jar,\
${HIVE_AUX_JARS_PATH}
fi
"""


was (Author: dili):
Hello Eric and Attila,

_hbase_lib_ was removed in BI 4.2.5. We should not use it post migration. The following section
in hive-env from IOP 4.2 should be removed during EU.

{quote}# Set HIVE_AUX_JARS_PATH
export HIVE_AUX_JARS_PATH={{hbase_lib}}/hbase-client.jar,\
{{hbase_lib}}/hbase-common.jar,\
{{hbase_lib}}/hbase-hadoop2-compat.jar,\
{{hbase_lib}}/hbase-prefix-tree.jar,\
{{hbase_lib}}/hbase-protocol.jar,\
{{hbase_lib}}/hbase-server.jar,\
{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\
${HIVE_AUX_JARS_PATH}
{quote}

Right around the time of the IOP/HDP annoucement, Tim,  myself, and a dev from our Hive team
were looking at a fix and our thinking was the following 

{quote}<service name="HIVE">
      <component name="HIVE_SERVER">
        <changes>
          <definition xsi:type="configure" id="biginsights_4_2_5_update_hive_env">
            <type>hive-env</type>
            <replace key="content" find="{{hbase_lib}}/hbase-client.jar," replace-with=""/>
            <replace key="content" find="{{hbase_lib}}/hbase-common.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-hadoop2-compat.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-prefix-tree.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-protocol.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/hbase-server.jar,\" replace-with="\"/>
            <replace key="content" find="{{hbase_lib}}/htrace-core-3.1.0-incubating.jar,\"
replace-with="\"/>
          </definition>{quote}

So the end result is like the following
{quote}export HIVE_AUX_JARS_PATH=\
\
\
\
\
\
\
${HIVE_AUX_JARS_PATH}{quote}
		  
*IMPORTANT*: Please note that for a BigSQL cluster to be migrated, hive-env has the following
at the end of the file that *must be retained during EU*. -- the section only exists on a
BigSQL cluster.

{quote}# Allow Hive to read Big SQL HBase tables
if [ -d "/usr/ibmpacks/current/bigsql/bigsql/lib/java" ]; then
export HIVE_AUX_JARS_PATH=\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-io.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-hbase.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/commoncatalog.jar,\
/usr/ibmpacks/current/bigsql/hive/lib/hive-hbase-handler.jar,\
${HIVE_AUX_JARS_PATH}
fi{quote}

P.S
I have an IOP4.2.5/BigSQL cluster migrated to HDP 2.6.2 and I am able to open the Hive shell
after I switch (the migration was done before Attila's EU config fix) the engine to MR.

[hive@loch4 ~]$ hive
2017-08-04 06:40:32,485 WARN  [main] conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce
does not exist
Logging initialized using configuration in file:/etc/hive/2.6.2.0-129/0/hive-log4j.properties
hive>

Here is what its hive aux lib setting looks like - the bigsql section at the end

{quote}
export HIVE_AUX_JARS_PATH="{{stack_root}}/current/ext/hive"

if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
  if [ -f "${HIVE_AUX_JARS_PATH}" ] || [ -d "${HIVE_AUX_JARS_PATH}" ] ; then
    export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
  elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
    export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar
  fi
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
  export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar
fi

export METASTORE_PORT={{hive_metastore_port}}

{% if sqla_db_used or lib_dir_available %}
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:{{jdbc_libs_dir}}"
export JAVA_LIBRARY_PATH="$JAVA_LIBRARY_PATH:{{jdbc_libs_dir}}"
{% endif %}

# Allow Hive to read Big SQL HBase tables
if [ -d "/usr/ibmpacks/current/bigsql/bigsql/lib/java" ]; then
export HIVE_AUX_JARS_PATH=\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-io.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/biga-hbase.jar,\
/usr/ibmpacks/current/bigsql/bigsql/lib/java/commoncatalog.jar,\
/usr/ibmpacks/current/bigsql/hive/lib/hive-hbase-handler.jar,\
${HIVE_AUX_JARS_PATH}
fi
{quote}

> Hive can not write to session directory after upgrade
> -----------------------------------------------------
>
>                 Key: AMBARI-21607
>                 URL: https://issues.apache.org/jira/browse/AMBARI-21607
>             Project: Ambari
>          Issue Type: Bug
>          Components: stacks
>    Affects Versions: 2.5.2
>            Reporter: Eric Yang
>            Assignee: Doroszlai, Attila
>             Fix For: 2.5.2
>
>
> After upgrade, start hive shell.  This error message is shown:
> {code}
> Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Previous
writer likely failed to write hdfs://eyang-1.openstacklocal:8020/tmp/hive/hive/_tez_session_dir/6a2cab98-2e71-4223-bc9d-6301b3c6a5d5/hbase-client.jar.
Failing because I am unlikely to write too.
> 	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:560)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: java.io.IOException: Previous writer likely failed to write hdfs://eyang-1.openstacklocal:8020/tmp/hive/hive/_tez_session_dir/6a2cab98-2e71-4223-bc9d-6301b3c6a5d5/hbase-client.jar.
Failing because I am unlikely to write too.
> 	at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1002)
> 	at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:882)
> 	at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:809)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:258)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:157)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
> 	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> 	... 8 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message