hdt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srimanth Gunturi <srima...@gmail.com>
Subject Re: Merging Hadoop-Eclipse project into HDT
Date Tue, 09 Jul 2013 07:35:05 GMT
Hi Rahul,
Hadoop-Eclipse UI is decoupled from models and controllers.
EMF models automatically provide model and controller separation.
That work is already done. If we reused the available code as is, we do not
have to rewrite a major bulk of the functionality.

IMHO, the least amount of work in getting both projects merged, is putting
effort into moving MR core/UI functionality onto the above plugins
following the same paradigms. If we did this, we wouldnt have to rewrite
HDFS, and the underlying internals/UI, as they are already working.
Best regards,
Srimanth




On Mon, Jul 8, 2013 at 11:02 PM, Rahul Sharma <rahul0208@gmail.com> wrote:

> +1 to the idea to have some abstraction to access HDFS/Zookeeper. We could
> make different implementations for different versions. As for complete UI
> decoupling , I think we would like to achieve it.  My only concern here is
> that this looks like complete overwrite as the current version UI is
> tightly coupled with logic. We should try to distribute this in multiple
> releases. I will dive into hadoop-eclipse some time this week and share my
> thoughts on the same J
>
>
> regards,
> Rahul
>
>
> On Mon, Jul 8, 2013 at 11:49 AM, Srimanth Gunturi <srimanth@gmail.com
> >wrote:
>
> >  Hello,
> > With regards to contributing HDFS/ZooKeeper functionalities, I was going
> > through HDT code and noted some design issues/thoughts that I wanted to
> > discuss.
> >
> > 1) Client cannot connect with multiple versions of HDFS/MR servers
> > org.apache.hdt.core.cluster.HadoopCluster which represents a cluster,
> > provides direct access to the HDFS and MR java API. This implies that at
> > any time, only 1 version of HDFS and MR client libraries can be used
> > (typically, whichever version gets loaded first by the classloader). So
> if
> > there was any use case where interactions with multiple HDFS/MR versions
> is
> > required, we would hit runtime issues. The client would be at the mercy
> of
> > backward/forward compatibility capabilities of the HDFS/MR/[any-service]
> > clients.
> >
> > In the Hadoop-Eclipse project, to get around this issue, I have created
> an
> > extension-point based abstraction, where the Eclipse functionality itself
> > would never directly use HDFS/ZooKeeper/[service] classes. Rather, from
> > multiple versions of extension point implementations, the right one would
> > be used to talk to the server. This allows the UI/core(headless)
> > functionalities to be free from the ever changing versions of
> > clients/servers.
> >
> >
> > 2) No clean seperation of UI, non-UI capabilities.
> > In HDFS, almost all functionality is non-UI (create, read, write, delete
> of
> > files/folders). However, currently all HDT plugins are dependent on UI
> > plugins (starting with org.apache.hdt.core). This goes against the
> > model-view-controller (MVC) paradigm, where the Eclipse UI (view) is
> mixed
> > in with the models and controllers. There is no reason why someone could
> > not leverage or extend the core/headless/non-UI capabilities of various
> > Hadoop services in Eclipse without the UI.
> >
> > In the Hadoop-Eclipse project, plugins are categorized into core
> > (representing non-UI capabilities) and UI plugins. You can create
> > connections, create/read/write/delete HDFS/ZooKeeper contents, etc.,
> > without even having UI plugins. This is helpful in nightly JUnit tests to
> > start. But it also allows others to provide their own UI interactions on
> > top of us. The models that are persisted (HDFS/ZooKeeper connections,
> > metadata, etc.) are Eclipse Modeling Framework
> > (EMF)<http://www.eclipse.org/modeling/emf/>models, which have a
> > built-in notification mechanism. They help in a clean
> > separation of Models and Controllers in MVC.
> >
> >
> > The above were some of the major ones which came to mind.
> > I encourage the community to go through the
> > Hadoop-Eclipse<http://people.apache.org/~srimanth/hadoop-eclipse/
> >project
> > codebase, and discuss any issues/concerns you have.
> >
> > I am thinking of the best way to merge the functionalities of both
> > projects, and would like to put forward a proposal.
> > HDFS is the only functionality common between both projects, along with
> > underlying framework. If we can come to a consensus on which parts we
> want
> > from where, it will be a smoother effort merging the code. From my end of
> > the spectrum, I was thinking it might be easier if the MR functionality
> > could be merged into the HDFS/ZooKeeper functionalities, thus providing a
> > union of both projects.
> >
> > I just wanted to get the merging process started, and look forward to
> > discussing more about it.
> > Best regards,
> > Srimanth
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message