oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imesha Sudasingha <imesha...@cse.mrt.ac.lk>
Subject Re: GSoC 2017 - Rework OODT Configuration to make use of Zookeeper
Date Thu, 16 Mar 2017 12:21:38 GMT
Hi Tom,

Thanks for the quick reply. I came up with a small design [1].

What I suggest is a separate module for configuration management, which can
be extended in future and used by many components as well.

As shown in the design [1], File Manager (or any component that is willing
to have this extended functionality of distributed configuration
management) will have an instance of *ConfigurationManager* which is an
interface defined to load configuration (specifically, .properties files)
and get configurations of each running instance of the same component (If
distributed configuration management is enabled, this will return the
configuration of all the other instances).

There will be two implementing classes of that interface,
*StandaloneConfigurationManager* and *DistributedConfigurationManager.* As
the name implies, standalone configuration manager is to be used if the
users do not intend to used distributed configuration management. Also this
will be same as the current mechanism of configuration management. Then the
distributed configuration manager will be the implementation using
Zookeeper.

At zookeeper level, we will need to have a proper naming mechanism and a
structure to store configuration. For example,*
/${nodeName}/${componentName}/{$configFileName}* will be the place where
the configuration file contents will be stored. *getFile(String fileName)
and getFiles(String directiryName)* of *NodeConfiguration* class will
return the content of the configuration files and configuration directories
respectively. I have demonstrated the ZNode structre I'm expecting to
design in [2]

Please note that this is just a very high level overview of my design and
the real difficulty of implementation comes when testing this and
addressing the edge cases of zookeeper. Also, I have only added few methods
that came to my mind to the *ConfigurationManager* interface. If you find
any fault in this design or have a better design in mind please let me know
so that I can optimize this design and do my best to come up with the best
possible solution.

Thank you!

[1] https://drive.google.com/open?id=0B415Bnipaf1ldnV1dWt4VVVLam8
[2] https://drive.google.com/open?id=0B415Bnipaf1lNThKYXZMN3pFemc

Kind Regards,
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and  Engineering,
University of Moratuwa,
Sri Lanka.

<https://lk.linkedin.com/in/imeshasudasingha>  <https://github.com/IMS94>
<http://stackoverflow.com/users/4012073/imesha-sudasingha>
<https://twitter.com/Imesha94>

On 16 March 2017 at 16:55, Tom Barber <tom.barber@meteorite.bi> wrote:

> Hi Imesha,
>
> Basically, just allowing the dynamic registration of file managers so that
> new ones can come and go.
>
> Tom
>
> On Thu, Mar 16, 2017 at 10:00 AM, Imesha Sudasingha <
> imesha.13@cse.mrt.ac.lk
> > wrote:
>
> > Hi Tom,
> >
> > Need a small clarification. In the last reply, you mentioned that file
> > managers may be designed to be *transient. *What do you mean by that? A
> > quick response would really help me as I'm trying to come up with a
> design.
> >
> > Thank you!
> >
> > *Imesha Sudasingha*
> > Undergraduate of Department of Computer Science and  Engineering,
> > University of Moratuwa,
> > Sri Lanka.
> >
> > <https://lk.linkedin.com/in/imeshasudasingha>  <https://github.com/IMS94
> >
> > <http://stackoverflow.com/users/4012073/imesha-sudasingha>
> > <https://twitter.com/Imesha94>
> >
> > On 13 March 2017 at 22:03, Imesha Sudasingha <imesha.13@cse.mrt.ac.lk>
> > wrote:
> >
> > > Hi Tom,
> > >
> > > Thanks for the clarification. It really helped a lot to fill most of
> the
> > > gaps I had unanswered. I have a draft design in my mind and will post
> for
> > > your review with details soon.
> > >
> > > As for my current understanding, my task is to implement a mechanism to
> > > switch/chose configuration mechanism (between local files and zookeer
> for
> > > the moment) and then allow APIs to query those mechanisms (in the
> > zookeeper
> > > scenario) for future use (For File Managers to know about other
> instances
> > > and make decisions accordingly). That include synchronization, fetching
> > > configuration from zookeeper if an instance go down and come again and
> so
> > > on. Am I correct?
> > >
> > > I will try to come up with the design I mentioned, so that both of us
> can
> > > discuss more about the improvements and requirements soon.
> > >
> > > Thank you for the support!
> > >
> > > Kind Regards,
> > > *Imesha Sudasingha*
> > > Undergraduate of Department of Computer Science and  Engineering,
> > > University of Moratuwa,
> > > Sri Lanka.
> > >
> > > <https://lk.linkedin.com/in/imeshasudasingha>  <
> https://github.com/IMS94
> > >
> > > <http://stackoverflow.com/users/4012073/imesha-sudasingha>
> > > <https://twitter.com/Imesha94>
> > >
> > > On 13 March 2017 at 20:19, Tom Barber <tom.barber@meteorite.bi> wrote:
> > >
> > >> Hi Imesha
> > >>
> > >> The best way to demonstrate this is to show you a reference build for
> > >> OODT..
> > >>
> > >> https://dl.dropboxusercontent.com/u/8503756/oodt-example.tgz
> > >>
> > >> If you can grab that archive and extract it, you'll see that is how
> > >> currently OODT is deployed.
> > >>
> > >> If you take the file manager as the reference module inside
> oodt/filemgr
> > >> you'll see
> > >>
> > >> etc/filemgr.properties
> > >>
> > >> and
> > >>
> > >> policy/oodt/*
> > >>
> > >> These are the configuration locations of a single file manager.
> > >>
> > >> The problem I want to solve is this:
> > >>
> > >> a) If I run > 1 file manager they are completely separate and they
> don't
> > >> know of each other. So using Zookeeper will allow multiple file
> managers
> > >> to
> > >> register against ZK and learn of each other.
> > >>
> > >> b) Backing them onto ZK will allow file managers to come and go on the
> > >> fly,
> > >> providing scaling capabilities or maybe fm's that are designed to be
> > >> transient.
> > >>
> > >> Just for clarity, the ZK configuration will be an optional
> requirement,
> > so
> > >> we need to extend the existing configuration mechanism, to support
> both
> > >> the
> > >> current config and the ZK config instead of  ZK being mandatory.
> > >>
> > >> Tom
> > >>
> > >> On Mon, Mar 13, 2017 at 3:20 AM, Imesha Sudasingha <
> > >> imesha.13@cse.mrt.ac.lk>
> > >> wrote:
> > >>
> > >> > Hi Tom,
> > >> >
> > >> > Related to the conversation on JIRA [1] ;
> > >> >
> > >> > I went through the CAS-File manager module and I now understand how
> to
> > >> > address ".properties" files. Since you mentioned in JIRA [2] that
> > there
> > >> are
> > >> > occasions where we use *spring configuration files *for
> configuration
> > >> > purposes, I want to have a look at that type of configuration too.
> Can
> > >> you
> > >> > give me an example module where spring configuration files are used
> to
> > >> load
> > >> > configuration?
> > >> >
> > >> > Also, will the content of a configuration file exceed 1MB (Maximum
> > data
> > >> > size allowed in an ZNode) in any case?
> > >> >
> > >> > Thank you!
> > >> >
> > >> > [1]
> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen
> > >> tId=15906204&
> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels:
> > >> > comment-tabpanel#comment-15906204
> > >> >
> > >> > [2]
> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen
> > >> tId=15906524&
> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels:
> > >> > comment-tabpanel#comment-15906524
> > >> >
> > >> > Kind Regards,
> > >> > *Imesha Sudasingha*
> > >> > Undergraduate of Department of Computer Science and  Engineering,
> > >> > University of Moratuwa,
> > >> > Sri Lanka.
> > >> >
> > >> > <https://lk.linkedin.com/in/imeshasudasingha>  <
> > >> https://github.com/IMS94>
> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha>
> > >> > <https://twitter.com/Imesha94>
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message