hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18953) Use hadoop shaded jars for hadoop-3 profile
Date Thu, 05 Oct 2017 21:32:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193756#comment-16193756

Sean Busbey commented on HBASE-18953:

Copying my comment from HADOOP-14178:

Maybe in the future? Right now HBase's dependency on Hadoop is kind of messy for a few reasons.
# we have to keep working on top of Hadoop 2.x and Hadoop 3.x. We mostly have this abstracted.
# we have parts of HBase that make use of Hadoop internals such that we can't currently move
all of the Hadoop-3 build version on to the client artifacts
# The parts of HBase that use Hadoop internals are currently all mixed up with parts that
are proper downstream consumers, so we can't even e.g. isolate the problem parts and then
avoid mockito there.

The hadoop client artifacts have to be used to the exclusion of the non-downstream facing
ones. you can't leave in e.g. hadoop-common. I have a branch sitting around somewhere from
the last time I tried to do this via either moving our internal Hadoop usage to different
modules or copying source into our tree. It was a morass, but things are considerably better
now in our codebase.

If you're interested in digging into this [~tedyu], I'd be willing to rebase my prior attempt
and note where it looks broken to give you a starting point. Fair warning though, it's probably
still a good deal of work.


> Use hadoop shaded jars for hadoop-3 profile
> -------------------------------------------
>                 Key: HBASE-18953
>                 URL: https://issues.apache.org/jira/browse/HBASE-18953
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>         Attachments: 18953.v1.txt
> Over HADOOP-14178 , [~bharatviswa] suggested using hadoop shaded jars for hadoop-3 since
Hadoop client run time has all 3rd party dependencies of Hadoop.
> This would avoid issues starting mini dfs cluster.

This message was sent by Atlassian JIRA

View raw message