hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haohui Mai <h...@hortonworks.com>
Subject Re: Putting the hdfs client as a separate jar
Date Fri, 04 Apr 2014 18:39:36 GMT
Tuning the POM only mitigates the problem. The problem of one HDFS jar is
that you can't rule out all unnecessary dependency. For example,
NamenodeWebHdfsMethods depends on jersey-server and servlet. The Apache
Falcon project has clients for HDFS, Hive, Pig, Oozie, thus it pulls in the
dependency. It also pulls in Tomcat as well.

Here is the bad part: the dependency from the client and Tomcat results in
three different versions of servlet jars in the classpath. They have the
same package name but they obviously have different behavior. We have to be
extremely careful on the orders of the jars in the classpath to make things
work.

One might be able to hand craft a hdfs-client POM to do the trick, but it
requires a lot of tuning and testing, and sometimes it might not be
feasible due to the dependency between the classes in the jar.

Pulling out the client to another module seems much cleaner and easier to
maintain.

~Haohui



On Thu, Apr 3, 2014 at 5:06 AM, Steve Loughran <stevel@hortonworks.com>wrote:

> to follow up with an example,
>
> JIRA on updating dependencies and tuning the POMs
> https://issues.apache.org/jira/browse/HADOOP-9991
>
>
>  here's a JIRA on dropping ZK from the hadoop-client POM
>
> https://issues.apache.org/jira/browse/HADOOP-9905
> 
> And there's an mr-client POM where we've been slowly cutting down on what
> it pulls in
> https://issues.apache.org/jira/browse/MAPREDUCE-5624
>
> This shows that
> 1. we can given maven/ivy projects what they need -and no more- through
> POM-only projects.
> 2. its an ongoing project to keep those dependencies cut down.
> 3. there's always the risk that you drop too much and some project
> discovers that while their code builds, it doesn't run any more.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message