hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-13016) reinstate hadoop-hdfs as dependency of hadoop-client, create hadoop-lean-client for minimal deployments
Date Mon, 11 Apr 2016 10:45:25 GMT
Steve Loughran created HADOOP-13016:
---------------------------------------

             Summary: reinstate hadoop-hdfs as dependency of hadoop-client, create hadoop-lean-client
for minimal deployments
                 Key: HADOOP-13016
                 URL: https://issues.apache.org/jira/browse/HADOOP-13016
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
    Affects Versions: 2.8.0
            Reporter: Steve Loughran


the split of hadoop-hdfs and hadoop-hdfs-client is breaking code of mine whose builds declared
a dependency on hadoop-client and expected all of HDFS to make it in.

I'm finding this first, because I'm building and testing downstream code against branch-2;
I find myself having to explicitly declare a dependency on hadoop-hdfs to make things work
again.

We've also seen problems downstream (e.g. spark) where the move of s3n classes to hadoop-aws
has broken code which expects it to be there.

At the same time, I see the merits in a lean, low-dependency client, which hadoop-client and
its dependencies is not today.

I propose

# reinstate hadoop-hdfs as dependency of hadoop-client
# add hadoop-aws as a dependency of hadoop-client —but excluding adding any amazon-aws JARs.
# create hadoop-lean-client for minimal deployments, stripping out all extraneous dependencies,
# for hadoop-lean-client, have a compatibility statement of "we will strip out anything we
can from this, even over point releases". That is, anything that can be dropped in future,
will.

This will give downstream projects a choice: the old POM with everything, the lean POM for
new apps.

And, by reinstating hadoop-hdfs, things will build again




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message