hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roman Shaposhnik (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-5286) bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation
Date Sat, 03 Mar 2012 07:15:59 GMT

     [ https://issues.apache.org/jira/browse/HBASE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Roman Shaposhnik updated HBASE-5286:
------------------------------------

    Attachment: HBASE-5286.patch.txt

Attaching the patch that covers the most generic case I could possibly think of. This patch
assumes the following:
  # HADOOP_HOME has precedence over HADOOP_PREFIX and both have precedence over the hadoop
that is available via PATH
  # installation of Hadoop is expected to have working bin/hadoop

I think this change gives us the most flexibility in supporting all sorts of versions of Hadoop
without trying to second guess their layout, etc. We only expect them to be functional ;-)

Please let me know if this can be included in 0.92.1
                
> bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented
with split packaged Hadoop 0.23 installation
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-5286
>                 URL: https://issues.apache.org/jira/browse/HBASE-5286
>             Project: HBase
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.92.0
>            Reporter: Roman Shaposhnik
>            Assignee: Roman Shaposhnik
>         Attachments: HBASE-5286.patch.txt
>
>
> Here's the bit from bin/hbase that might need TLC now that Hadoop can be spotted in the
wild in split-package configuration:
> {noformat}
> #If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
> if [ ! -z $HADOOP_HOME ]; then
>   HADOOPCPPATH=""
>   if [ -z $HADOOP_CONF_DIR ]; then
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_HOME}/conf")
>   else
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_CONF_DIR}")
>   fi
>   if [ "`echo ${HADOOP_HOME}/hadoop-core*.jar`" != "${HADOOP_HOME}/hadoop-core*.jar"
] ; then
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-core*.jar
| head -1`)
>   else
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-common*.jar
| head -1`)
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-hdfs*.jar
| head -1`)
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-mapred*.jar
| head -1`)
>   fi
> {noformat}
> There's a couple of issues with the above code:
>    0. HADOOP_HOME is now deprecated in Hadoop 0.23
>    1. the list of jar files added to the class-path should be revised
>    2. we need to figure out a more robust way to get the jar files that are needed to
the classpath (things like hadoop-mapred*.jar tend to match src/test jars as well)
> Better yet, it would be useful to look into whether we can transition HBase's bin/hbase
onto using bin/hadoop as a launcher script instead of direct JAVA invocations (Pig, Hive,
Sqoop and Mahout already do that)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message