hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roman Shaposhnik (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5286) bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation
Date Mon, 20 Feb 2012 23:59:34 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212229#comment-13212229
] 

Roman Shaposhnik commented on HBASE-5286:
-----------------------------------------

@Todd,

I see 2 reasons to potentially prefer the full-blown execution script:
  # it is guaranteed to work (while hadoop classpath could, potentially, have bugs)
  # it takes care of niatve libraries setting (and anything else for that matter that might
not be covered by the CP)

Of course, the last point is way stronger than the first one.

That said, I think making HBase script use hadoop classpath instead of the current logic definitely
qualifies as an incremental improvement. I'll submit the patch tomorrow and we can worry about
native libraries, etc later.
                
> bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented
with split packaged Hadoop 0.23 installation
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-5286
>                 URL: https://issues.apache.org/jira/browse/HBASE-5286
>             Project: HBase
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.92.0
>            Reporter: Roman Shaposhnik
>            Assignee: Roman Shaposhnik
>
> Here's the bit from bin/hbase that might need TLC now that Hadoop can be spotted in the
wild in split-package configuration:
> {noformat}
> #If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
> if [ ! -z $HADOOP_HOME ]; then
>   HADOOPCPPATH=""
>   if [ -z $HADOOP_CONF_DIR ]; then
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_HOME}/conf")
>   else
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_CONF_DIR}")
>   fi
>   if [ "`echo ${HADOOP_HOME}/hadoop-core*.jar`" != "${HADOOP_HOME}/hadoop-core*.jar"
] ; then
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-core*.jar
| head -1`)
>   else
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-common*.jar
| head -1`)
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-hdfs*.jar
| head -1`)
>     HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-mapred*.jar
| head -1`)
>   fi
> {noformat}
> There's a couple of issues with the above code:
>    0. HADOOP_HOME is now deprecated in Hadoop 0.23
>    1. the list of jar files added to the class-path should be revised
>    2. we need to figure out a more robust way to get the jar files that are needed to
the classpath (things like hadoop-mapred*.jar tend to match src/test jars as well)
> Better yet, it would be useful to look into whether we can transition HBase's bin/hbase
onto using bin/hadoop as a launcher script instead of direct JAVA invocations (Pig, Hive,
Sqoop and Mahout already do that)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message