Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of hmai@hortonworks.com
 designates 209.85.160.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1396475321.69501.YahooMailNeo@web125704.mail.ne1.yahoo.com>
References: 
 <CABbyifMYXSbDtpS-q6Z0oRubgtDCe_NRqdZiM8xkmb0bV_u_QA@mail.gmail.com>
	<1396475321.69501.YahooMailNeo@web125704.mail.ne1.yahoo.com>
Date: Wed, 2 Apr 2014 15:02:53 -0700
Message-ID: 
 <CABbyifMfu_kMAAn8wX4buYVGhktLRnKw7gY7vrd0utphCY2+Sw@mail.gmail.com>
Subject: Re: Putting the hdfs client as a separate jar
From: Haohui Mai <hmai@hortonworks.com>
To: hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a1136bd6644216f04f61672da

--001a1136bd6644216f04f61672da
Content-Type: text/plain; charset=US-ASCII

The rpc and the web client can stay in one jar for the first cut. Indeed it
might introduce some extra dependency, but the downstream projects always
have the option to implement the webhdfs protocol themselves if they really
need to avoid the dependency.

Hadoop common is a bigger problem. Indeed the hadoop common jar needs to be
separated into smaller modules to minimize the dependency. This needs to be
addressed as well.

The good news is that it can be done in a incremental way. We can revisit
the dependency of hadoop-common after separating the jar of the hdfs client.

~Haohui


On Wed, Apr 2, 2014 at 2:48 PM, Tsz Wo Sze <szetszwo@yahoo.com> wrote:

> It is a very good idea although it might not be easy to do.  One aspect to
> consider is that do we need separated jars for rpc client and web client?
>  Now, suppose we could successfully separate HFDS Client jar(s) from HDFS.
>  However, HDFS Client uses Common as a library.  We have to
> separate Common since it also has a lot of dependent jars.  I guess we
> might have to divide Common and HDFS into small modules and figure out the
> dependency between them.
>
> Tsz-Wo
> On Wednesday, April 2, 2014 2:28 PM, Haohui Mai <hmai@hortonworks.com>
> wrote:
>
> Hi,
> >
> >Many downstream projects needs the ability to access hdfs. In order to do
> >this, currently downstream projects are forced to bring in the whole hdfs
> >jar and its dependency, since both the hdfs server and the hdfs client
> >reside in the same jar.
> >
> >To integrate with hdfs, the downstream projects are forced to manage the
> >excess dependency from the hdfs server side (e.g., jersey, servlet, netty,
> >and jsp-runtime, just to name a few). In my own experience, I ended up
> >spending quite a bit of time on tweaking the POMs to work around
> collisions
> >of dependent jars.
> >
> >To solve this problem, I propose to reorganize the code to put hdfs client
> >into a separate jar. That way the client jar no longer depends on the jars
> >that are required by the server side, therefore it is easier for the
> >downstream projects to integrate with hdfs.
> >
> >Your feedbacks are appreciated.
> >
> >Regards,
> >Haohui
> >
> >--
> >CONFIDENTIALITY NOTICE
> >NOTICE: This message is intended for the use of the individual or entity
> to
> >which it is addressed and may contain information that is confidential,
> >privileged and exempt from disclosure under applicable law. If the reader
> >of this message is not the intended recipient, you are hereby notified
> that
> >any printing, copying, dissemination, distribution, disclosure or
> >forwarding of this communication is strictly prohibited. If you have
> >received this communication in error, please contact the sender
> immediately
> >and delete it from your system. Thank You.
> >
> >
> >

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

--001a1136bd6644216f04f61672da--