hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haohui Mai <h...@hortonworks.com>
Subject Re: Putting the hdfs client as a separate jar
Date Fri, 04 Apr 2014 18:46:48 GMT
I agree with Nicholas, Steve and Alejandro that it might require some
nontrivial to achieve the goal. Here is my high-level plan:

1. Create a new hdfs-client package, and gradually move classes from hdfs
to hdfs-client. Fortunately IDEs like Eclipse and IntelliJ can do most of
the heavy-liftings.
2. The hdfs package depends on the hdfs-client, because the webhdfs server
uses a DFSClient internally.
3. Downstream projects can continue to depend on hdfs in the meantime, but
they can start using the new hdfs-client projects once we make enough
progress.

That way the work can be done incrementally. Tackling the dependency of
hadoop-common requires more work thus I plan to take care of it after we
have a separate hdfs-client. Does it sound a reasonable plan?

~Haohui


On Thu, Apr 3, 2014 at 10:16 AM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> Haouhi's suggestion of a hdfs-client JAR with client dependencies only,
> would be IMO the 'correct' way of doing things, we should have a
> hdfs-server and hdfs-client JARs.
>
> Doing this is practice is not trivial as classes are not properly
> segregated. So, Steven's suggestion of  an hdfs-client seems the best bet
> short term.
>
> thx
>
>
> On Thu, Apr 3, 2014 at 5:06 AM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > to follow up with an example,
> >
> > JIRA on updating dependencies and tuning the POMs
> > https://issues.apache.org/jira/browse/HADOOP-9991
> >
> >
> >  here's a JIRA on dropping ZK from the hadoop-client POM
> >
> > https://issues.apache.org/jira/browse/HADOOP-9905
> >
> > And there's an mr-client POM where we've been slowly cutting down on what
> > it pulls in
> > https://issues.apache.org/jira/browse/MAPREDUCE-5624
> >
> > This shows that
> > 1. we can given maven/ivy projects what they need -and no more- through
> > POM-only projects.
> > 2. its an ongoing project to keep those dependencies cut down.
> > 3. there's always the risk that you drop too much and some project
> > discovers that while their code builds, it doesn't run any more.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message