hdt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mirko.kaempf" <mirko.kae...@gmail.com>
Subject AW: Re: Merging Hadoop-Eclipse project into HDT
Date Fri, 12 Jul 2013 05:46:07 GMT
+1 for working with a feature branch



Von Samsung Mobile gesendet

-------- Ursprüngliche Nachricht --------
Von: Adam Berry <adamb@apache.org> 
Datum: 11.07.2013  22:51  (GMT+00:00) 
An: dev@hdt.incubator.apache.org 
Betreff: Re: Merging Hadoop-Eclipse project into HDT 
 
Hey guys,

so for background, the intent for this project was always to get to where
we can connect to multiple versions of Hadoop clusters from one dev tools
install. http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposalfor
a little reference.

The code as it stands out is basically a first pass at splitting the code
from Hadoop contrib, there it resided as a single plug-in, with all the
same client library limitations. And yes, you cannot connect to multiple
versions of Hadoop etc, and the logic is pretty coupled in some spots.

Personally, I'm all for pulling in this new work to for HDFS and zookeeper
work, and then bringing the MR side up to the same point with the same
approach, as Srimanth suggested.

I also vote for doing this in a feature branch in our git repo, and if no
one objects I'll get that setup so we can get to work.

Cheers,
Adam


On Tue, Jul 9, 2013 at 2:35 AM, Srimanth Gunturi <srimanth@gmail.com> wrote:

> Hi Rahul,
> Hadoop-Eclipse UI is decoupled from models and controllers.
> EMF models automatically provide model and controller separation.
> That work is already done. If we reused the available code as is, we do not
> have to rewrite a major bulk of the functionality.
>
> IMHO, the least amount of work in getting both projects merged, is putting
> effort into moving MR core/UI functionality onto the above plugins
> following the same paradigms. If we did this, we wouldnt have to rewrite
> HDFS, and the underlying internals/UI, as they are already working.
> Best regards,
> Srimanth
>
>
>
>
> On Mon, Jul 8, 2013 at 11:02 PM, Rahul Sharma <rahul0208@gmail.com> wrote:
>
> > +1 to the idea to have some abstraction to access HDFS/Zookeeper. We
> could
> > make different implementations for different versions. As for complete UI
> > decoupling , I think we would like to achieve it.  My only concern here
> is
> > that this looks like complete overwrite as the current version UI is
> > tightly coupled with logic. We should try to distribute this in multiple
> > releases. I will dive into hadoop-eclipse some time this week and share
> my
> > thoughts on the same J
> >
> >
> > regards,
> > Rahul
> >
> >
> > On Mon, Jul 8, 2013 at 11:49 AM, Srimanth Gunturi <srimanth@gmail.com
> > >wrote:
> >
> > >  Hello,
> > > With regards to contributing HDFS/ZooKeeper functionalities, I was
> going
> > > through HDT code and noted some design issues/thoughts that I wanted to
> > > discuss.
> > >
> > > 1) Client cannot connect with multiple versions of HDFS/MR servers
> > > org.apache.hdt.core.cluster.HadoopCluster which represents a cluster,
> > > provides direct access to the HDFS and MR java API. This implies that
> at
> > > any time, only 1 version of HDFS and MR client libraries can be used
> > > (typically, whichever version gets loaded first by the classloader). So
> > if
> > > there was any use case where interactions with multiple HDFS/MR
> versions
> > is
> > > required, we would hit runtime issues. The client would be at the mercy
> > of
> > > backward/forward compatibility capabilities of the
> HDFS/MR/[any-service]
> > > clients.
> > >
> > > In the Hadoop-Eclipse project, to get around this issue, I have created
> > an
> > > extension-point based abstraction, where the Eclipse functionality
> itself
> > > would never directly use HDFS/ZooKeeper/[service] classes. Rather, from
> > > multiple versions of extension point implementations, the right one
> would
> > > be used to talk to the server. This allows the UI/core(headless)
> > > functionalities to be free from the ever changing versions of
> > > clients/servers.
> > >
> > >
> > > 2) No clean seperation of UI, non-UI capabilities.
> > > In HDFS, almost all functionality is non-UI (create, read, write,
> delete
> > of
> > > files/folders). However, currently all HDT plugins are dependent on UI
> > > plugins (starting with org.apache.hdt.core). This goes against the
> > > model-view-controller (MVC) paradigm, where the Eclipse UI (view) is
> > mixed
> > > in with the models and controllers. There is no reason why someone
> could
> > > not leverage or extend the core/headless/non-UI capabilities of various
> > > Hadoop services in Eclipse without the UI.
> > >
> > > In the Hadoop-Eclipse project, plugins are categorized into core
> > > (representing non-UI capabilities) and UI plugins. You can create
> > > connections, create/read/write/delete HDFS/ZooKeeper contents, etc.,
> > > without even having UI plugins. This is helpful in nightly JUnit tests
> to
> > > start. But it also allows others to provide their own UI interactions
> on
> > > top of us. The models that are persisted (HDFS/ZooKeeper connections,
> > > metadata, etc.) are Eclipse Modeling Framework
> > > (EMF)<http://www.eclipse.org/modeling/emf/>models, which have a
> > > built-in notification mechanism. They help in a clean
> > > separation of Models and Controllers in MVC.
> > >
> > >
> > > The above were some of the major ones which came to mind.
> > > I encourage the community to go through the
> > > Hadoop-Eclipse<http://people.apache.org/~srimanth/hadoop-eclipse/
> > >project
> > > codebase, and discuss any issues/concerns you have.
> > >
> > > I am thinking of the best way to merge the functionalities of both
> > > projects, and would like to put forward a proposal.
> > > HDFS is the only functionality common between both projects, along with
> > > underlying framework. If we can come to a consensus on which parts we
> > want
> > > from where, it will be a smoother effort merging the code. From my end
> of
> > > the spectrum, I was thinking it might be easier if the MR functionality
> > > could be merged into the HDFS/ZooKeeper functionalities, thus
> providing a
> > > union of both projects.
> > >
> > > I just wanted to get the merging process started, and look forward to
> > > discussing more about it.
> > > Best regards,
> > > Srimanth
> > >
> >
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message