drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mufy <mufeed.us...@gmail.com>
Subject Re: About Storage Plugin Instance Broadcast
Date Sat, 28 Jun 2014 12:12:58 GMT
So as an end-user where exactly do I add my new storage plugin
configurations if I choose to add them in the future?


---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>





On Sat, Jun 28, 2014 at 3:17 AM, Aditya <adityakishore@gmail.com> wrote:

> To be more precise, the configuration file "storage-plugins.json" has been
> replaced with "bootstrap-storage-plugins.json" which resides inside the
> Drill jars.
>
> When a cluster is started for the first time and no existing storage plugin
> definition is found in the Zookeeper, this file will be read and copied
> into the Zookeeper. From that point onward, the all Drillbits in the
> cluster will use only the information stored in the ZK and would ignore "
> bootstrap-storage-plugins.json" as well.
>
>
> On Fri, Jun 27, 2014 at 2:36 PM, Aditya <adityakishore@gmail.com> wrote:
>
> > The configuration file "storage-plugins.json" is no longer used by Drill.
> >
> > Any change in the plugin configuration is stored only in the persistent
> > store (Zookeeper by default).
> >
> >
> > On Fri, Jun 27, 2014 at 12:09 AM, mufy <mufeed.usman@gmail.com> wrote:
> >
> >> Hello Anil,
> >>
> >> Hmm... so the ZK will have the info. Agreed. But what about the
> statement
> >> where it says that any new storage plugin added/updated will be
> broadcast
> >> to all the nodes running drillbit service. I took that statement to
> mean -
> >> the config file will be synchronized across all the nodes running the
> >> drillbit service. Did I get it wrong?
> >>
> >>
> >> ---
> >> Mufeed Usman
> >> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> >> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> >> <http://mufeed.livejournal.com>
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <akumarb2010@gmail.com>
> >> wrote:
> >>
> >> > Hi Murfy,
> >> >
> >> > Based on my understanding, registered storage plugin info will be
> >> stored on
> >> > ZK. As below.
> >> >
> >> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
> >> > [hive, cp, dfs, hbase]
> >> > [zk: localhost:2181(CONNECTED) 11] get
> /drill/sys.storage_plugins/hbase
> >> > {
> >> >   "type" : "hbase",
> >> >   "config" : {
> >> >     "hbase.zookeeper.quorum" : "localhost",
> >> >     "hbase.zookeeper.property.clientPort" : "2181"
> >> >   },
> >> >   "enabled" : true
> >> > }
> >> > cZxid = 0x428
> >> > ctime = Tue Jun 24 13:30:18 IST 2014
> >> > mZxid = 0x450
> >> > mtime = Tue Jun 24 13:31:41 IST 2014
> >> > pZxid = 0x428
> >> > cversion = 0
> >> > dataVersion = 1
> >> > aclVersion = 0
> >> > ephemeralOwner = 0x0
> >> > dataLength = 157
> >> > numChildren = 0
> >> > [zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
> >> > {
> >> >   "type" : "file",
> >> >   "enabled" : true,
> >> >   "connection" : "classpath:///",
> >> >   "workspaces" : null,
> >> >   "formats" : null
> >> > }
> >> > cZxid = 0x426
> >> > ctime = Tue Jun 24 13:30:18 IST 2014
> >> > mZxid = 0x426
> >> > mtime = Tue Jun 24 13:30:18 IST 2014
> >> > pZxid = 0x426
> >> > cversion = 0
> >> > dataVersion = 0
> >> > aclVersion = 0
> >> > ephemeralOwner = 0x0
> >> > dataLength = 118
> >> > numChildren = 0
> >> > [zk: localhost:2181(CONNECTED) 13]
> >> >
> >> > Thanks,
> >> > B Anil Kumar.
> >> >
> >> > Thanks & Regards,
> >> > B Anil Kumar.
> >> >
> >> >
> >> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mufeed.usman@gmail.com>
> wrote:
> >> >
> >> > > Stated here
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> >> > > that,
> >> > >
> >> > > "When you add or update storage plugin instances on one node, Drill
> >> > > broadcasts the information to all of the other Drill nodes so they
> >> have
> >> > > identical storage plugin instances registered. You do not need to
> >> restart
> >> > > any of the Drillbits when you add or update a storage plugin
> >> instance."
> >> > >
> >> > > Does that mean if I were to have the following on one node:
> >> > > [root@mufeed-drill1 ~]# cat
> >> > >
> >> > >
> >> >
> >>
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> >> > > {
> >> > >   "storage": {
> >> > >
> >> > >     dfs: {
> >> > >       type: "file",
> >> > >       connection: "maprfs:///"
> >> > >     },
> >> > >
> >> > >     cp: {
> >> > >       type: "file",
> >> > >       connection: "classpath:///"
> >> > >     },
> >> > >
> >> > >     cp1: {
> >> > >       type: "file",
> >> > >       connection: "classpath:///"
> >> > >     },
> >> > >       }
> >> > > }
> >> > >
> >> > > The same content would get replicated to other nodes, on a file
> >> content
> >> > > level? If yes, I do not see that happening. If not, please clarify.
> >> > >
> >> > > ---
> >> > > Mufeed Usman
> >> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400>
|
> >> My
> >> > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> LiveJournal
> >> > > <http://mufeed.livejournal.com>
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message