oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imesha Sudasingha <imesha...@cse.mrt.ac.lk>
Subject Re: GSoC 2017 - Rework OODT Configuration to make use of Zookeeper
Date Sat, 18 Mar 2017 12:39:08 GMT
Hi Tom,

What do you think of the design? Do you agree on implementing this
configuration management section in a separate module?

I was thinking on starting on the implementation as a proof of concept. Can
you give me your kind feedback before I start on that?

Thank you!

Kind Regards,
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and  Engineering,
University of Moratuwa,
Sri Lanka.

<https://lk.linkedin.com/in/imeshasudasingha>  <https://github.com/IMS94>
<http://stackoverflow.com/users/4012073/imesha-sudasingha>
<https://twitter.com/Imesha94>

On 16 March 2017 at 17:51, Imesha Sudasingha <imesha.13@cse.mrt.ac.lk>
wrote:

> Hi Tom,
>
> Thanks for the quick reply. I came up with a small design [1].
>
> What I suggest is a separate module for configuration management, which
> can be extended in future and used by many components as well.
>
> As shown in the design [1], File Manager (or any component that is willing
> to have this extended functionality of distributed configuration
> management) will have an instance of *ConfigurationManager* which is an
> interface defined to load configuration (specifically, .properties files)
> and get configurations of each running instance of the same component (If
> distributed configuration management is enabled, this will return the
> configuration of all the other instances).
>
> There will be two implementing classes of that interface,
> *StandaloneConfigurationManager* and *DistributedConfigurationManager.* As
> the name implies, standalone configuration manager is to be used if the
> users do not intend to used distributed configuration management. Also this
> will be same as the current mechanism of configuration management. Then the
> distributed configuration manager will be the implementation using
> Zookeeper.
>
> At zookeeper level, we will need to have a proper naming mechanism and a
> structure to store configuration. For example,*
> /${nodeName}/${componentName}/{$configFileName}* will be the place where
> the configuration file contents will be stored. *getFile(String fileName)
> and getFiles(String directiryName)* of *NodeConfiguration* class will
> return the content of the configuration files and configuration directories
> respectively. I have demonstrated the ZNode structre I'm expecting to
> design in [2]
>
> Please note that this is just a very high level overview of my design and
> the real difficulty of implementation comes when testing this and
> addressing the edge cases of zookeeper. Also, I have only added few methods
> that came to my mind to the *ConfigurationManager* interface. If you find
> any fault in this design or have a better design in mind please let me know
> so that I can optimize this design and do my best to come up with the best
> possible solution.
>
> Thank you!
>
> [1] https://drive.google.com/open?id=0B415Bnipaf1ldnV1dWt4VVVLam8
> [2] https://drive.google.com/open?id=0B415Bnipaf1lNThKYXZMN3pFemc
>
> Kind Regards,
> *Imesha Sudasingha*
> Undergraduate of Department of Computer Science and  Engineering,
> University of Moratuwa,
> Sri Lanka.
>
> <https://lk.linkedin.com/in/imeshasudasingha>  <https://github.com/IMS94>
> <http://stackoverflow.com/users/4012073/imesha-sudasingha>
> <https://twitter.com/Imesha94>
>
> On 16 March 2017 at 16:55, Tom Barber <tom.barber@meteorite.bi> wrote:
>
>> Hi Imesha,
>>
>> Basically, just allowing the dynamic registration of file managers so that
>> new ones can come and go.
>>
>> Tom
>>
>> On Thu, Mar 16, 2017 at 10:00 AM, Imesha Sudasingha <
>> imesha.13@cse.mrt.ac.lk
>> > wrote:
>>
>> > Hi Tom,
>> >
>> > Need a small clarification. In the last reply, you mentioned that file
>> > managers may be designed to be *transient. *What do you mean by that? A
>> > quick response would really help me as I'm trying to come up with a
>> design.
>> >
>> > Thank you!
>> >
>> > *Imesha Sudasingha*
>> > Undergraduate of Department of Computer Science and  Engineering,
>> > University of Moratuwa,
>> > Sri Lanka.
>> >
>> > <https://lk.linkedin.com/in/imeshasudasingha>  <
>> https://github.com/IMS94>
>> > <http://stackoverflow.com/users/4012073/imesha-sudasingha>
>> > <https://twitter.com/Imesha94>
>> >
>> > On 13 March 2017 at 22:03, Imesha Sudasingha <imesha.13@cse.mrt.ac.lk>
>> > wrote:
>> >
>> > > Hi Tom,
>> > >
>> > > Thanks for the clarification. It really helped a lot to fill most of
>> the
>> > > gaps I had unanswered. I have a draft design in my mind and will post
>> for
>> > > your review with details soon.
>> > >
>> > > As for my current understanding, my task is to implement a mechanism
>> to
>> > > switch/chose configuration mechanism (between local files and zookeer
>> for
>> > > the moment) and then allow APIs to query those mechanisms (in the
>> > zookeeper
>> > > scenario) for future use (For File Managers to know about other
>> instances
>> > > and make decisions accordingly). That include synchronization,
>> fetching
>> > > configuration from zookeeper if an instance go down and come again
>> and so
>> > > on. Am I correct?
>> > >
>> > > I will try to come up with the design I mentioned, so that both of us
>> can
>> > > discuss more about the improvements and requirements soon.
>> > >
>> > > Thank you for the support!
>> > >
>> > > Kind Regards,
>> > > *Imesha Sudasingha*
>> > > Undergraduate of Department of Computer Science and  Engineering,
>> > > University of Moratuwa,
>> > > Sri Lanka.
>> > >
>> > > <https://lk.linkedin.com/in/imeshasudasingha>  <
>> https://github.com/IMS94
>> > >
>> > > <http://stackoverflow.com/users/4012073/imesha-sudasingha>
>> > > <https://twitter.com/Imesha94>
>> > >
>> > > On 13 March 2017 at 20:19, Tom Barber <tom.barber@meteorite.bi>
>> wrote:
>> > >
>> > >> Hi Imesha
>> > >>
>> > >> The best way to demonstrate this is to show you a reference build for
>> > >> OODT..
>> > >>
>> > >> https://dl.dropboxusercontent.com/u/8503756/oodt-example.tgz
>> > >>
>> > >> If you can grab that archive and extract it, you'll see that is how
>> > >> currently OODT is deployed.
>> > >>
>> > >> If you take the file manager as the reference module inside
>> oodt/filemgr
>> > >> you'll see
>> > >>
>> > >> etc/filemgr.properties
>> > >>
>> > >> and
>> > >>
>> > >> policy/oodt/*
>> > >>
>> > >> These are the configuration locations of a single file manager.
>> > >>
>> > >> The problem I want to solve is this:
>> > >>
>> > >> a) If I run > 1 file manager they are completely separate and they
>> don't
>> > >> know of each other. So using Zookeeper will allow multiple file
>> managers
>> > >> to
>> > >> register against ZK and learn of each other.
>> > >>
>> > >> b) Backing them onto ZK will allow file managers to come and go on
>> the
>> > >> fly,
>> > >> providing scaling capabilities or maybe fm's that are designed to be
>> > >> transient.
>> > >>
>> > >> Just for clarity, the ZK configuration will be an optional
>> requirement,
>> > so
>> > >> we need to extend the existing configuration mechanism, to support
>> both
>> > >> the
>> > >> current config and the ZK config instead of  ZK being mandatory.
>> > >>
>> > >> Tom
>> > >>
>> > >> On Mon, Mar 13, 2017 at 3:20 AM, Imesha Sudasingha <
>> > >> imesha.13@cse.mrt.ac.lk>
>> > >> wrote:
>> > >>
>> > >> > Hi Tom,
>> > >> >
>> > >> > Related to the conversation on JIRA [1] ;
>> > >> >
>> > >> > I went through the CAS-File manager module and I now understand
>> how to
>> > >> > address ".properties" files. Since you mentioned in JIRA [2] that
>> > there
>> > >> are
>> > >> > occasions where we use *spring configuration files *for
>> configuration
>> > >> > purposes, I want to have a look at that type of configuration
too.
>> Can
>> > >> you
>> > >> > give me an example module where spring configuration files are
>> used to
>> > >> load
>> > >> > configuration?
>> > >> >
>> > >> > Also, will the content of a configuration file exceed 1MB (Maximum
>> > data
>> > >> > size allowed in an ZNode) in any case?
>> > >> >
>> > >> > Thank you!
>> > >> >
>> > >> > [1]
>> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen
>> > >> tId=15906204&
>> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels:
>> > >> > comment-tabpanel#comment-15906204
>> > >> >
>> > >> > [2]
>> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen
>> > >> tId=15906524&
>> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels:
>> > >> > comment-tabpanel#comment-15906524
>> > >> >
>> > >> > Kind Regards,
>> > >> > *Imesha Sudasingha*
>> > >> > Undergraduate of Department of Computer Science and  Engineering,
>> > >> > University of Moratuwa,
>> > >> > Sri Lanka.
>> > >> >
>> > >> > <https://lk.linkedin.com/in/imeshasudasingha>  <
>> > >> https://github.com/IMS94>
>> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha>
>> > >> > <https://twitter.com/Imesha94>
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message