Mailing-List: contact drill-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: drill-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of adityakishore@gmail.com
 designates 209.85.160.178 as permitted sender)
MIME-Version: 1.0
Reply-To: adityakishore@gmail.com
In-Reply-To: 
 <CAP+2gDpEEXvfQhTKu09xn21A9o-BR7JFxoQ_At3ejBV_RFJ9JA@mail.gmail.com>
References: 
 <CA+eYymEJcopG+Pj0Ly=-DRCZ7d58qihDx2cc=cCv8ReA_QR_eA@mail.gmail.com>
	<CAHdEuM2Mz9X+YJmfctF-5_+LX5O37BfbfZPnoUJpO0Dw2ECGHg@mail.gmail.com>
	<CA+eYymEK13bLq50jRJV58bOzxFasjKfKoGn0BidXmguLK87k=w@mail.gmail.com>
	<CAP+2gDpEEXvfQhTKu09xn21A9o-BR7JFxoQ_At3ejBV_RFJ9JA@mail.gmail.com>
Date: Fri, 27 Jun 2014 14:47:23 -0700
Message-ID: 
 <CAP+2gDpgT89p5feubLCxmcGsaw2ZJ33NOARcLZfjpARcAf-mSg@mail.gmail.com>
Subject: Re: About Storage Plugin Instance Broadcast
From: Aditya <adityakishore@gmail.com>
To: "drill-user@incubator.apache.org" <drill-user@incubator.apache.org>
Content-Type: multipart/alternative; boundary=20cf303bf88e277e9f04fcd84198

--20cf303bf88e277e9f04fcd84198
Content-Type: text/plain; charset=UTF-8

To be more precise, the configuration file "storage-plugins.json" has been
replaced with "bootstrap-storage-plugins.json" which resides inside the
Drill jars.

When a cluster is started for the first time and no existing storage plugin
definition is found in the Zookeeper, this file will be read and copied
into the Zookeeper. From that point onward, the all Drillbits in the
cluster will use only the information stored in the ZK and would ignore "
bootstrap-storage-plugins.json" as well.


On Fri, Jun 27, 2014 at 2:36 PM, Aditya <adityakishore@gmail.com> wrote:

> The configuration file "storage-plugins.json" is no longer used by Drill.
>
> Any change in the plugin configuration is stored only in the persistent
> store (Zookeeper by default).
>
>
> On Fri, Jun 27, 2014 at 12:09 AM, mufy <mufeed.usman@gmail.com> wrote:
>
>> Hello Anil,
>>
>> Hmm... so the ZK will have the info. Agreed. But what about the statement
>> where it says that any new storage plugin added/updated will be broadcast
>> to all the nodes running drillbit service. I took that statement to mean -
>> the config file will be synchronized across all the nodes running the
>> drillbit service. Did I get it wrong?
>>
>>
>> ---
>> Mufeed Usman
>> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
>> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
>> <http://mufeed.livejournal.com>
>>
>>
>>
>>
>>
>> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <akumarb2010@gmail.com>
>> wrote:
>>
>> > Hi Murfy,
>> >
>> > Based on my understanding, registered storage plugin info will be
>> stored on
>> > ZK. As below.
>> >
>> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
>> > [hive, cp, dfs, hbase]
>> > [zk: localhost:2181(CONNECTED) 11] get /drill/sys.storage_plugins/hbase
>> > {
>> >   "type" : "hbase",
>> >   "config" : {
>> >     "hbase.zookeeper.quorum" : "localhost",
>> >     "hbase.zookeeper.property.clientPort" : "2181"
>> >   },
>> >   "enabled" : true
>> > }
>> > cZxid = 0x428
>> > ctime = Tue Jun 24 13:30:18 IST 2014
>> > mZxid = 0x450
>> > mtime = Tue Jun 24 13:31:41 IST 2014
>> > pZxid = 0x428
>> > cversion = 0
>> > dataVersion = 1
>> > aclVersion = 0
>> > ephemeralOwner = 0x0
>> > dataLength = 157
>> > numChildren = 0
>> > [zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
>> > {
>> >   "type" : "file",
>> >   "enabled" : true,
>> >   "connection" : "classpath:///",
>> >   "workspaces" : null,
>> >   "formats" : null
>> > }
>> > cZxid = 0x426
>> > ctime = Tue Jun 24 13:30:18 IST 2014
>> > mZxid = 0x426
>> > mtime = Tue Jun 24 13:30:18 IST 2014
>> > pZxid = 0x426
>> > cversion = 0
>> > dataVersion = 0
>> > aclVersion = 0
>> > ephemeralOwner = 0x0
>> > dataLength = 118
>> > numChildren = 0
>> > [zk: localhost:2181(CONNECTED) 13]
>> >
>> > Thanks,
>> > B Anil Kumar.
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>> >
>> >
>> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mufeed.usman@gmail.com> wrote:
>> >
>> > > Stated here
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
>> > > that,
>> > >
>> > > "When you add or update storage plugin instances on one node, Drill
>> > > broadcasts the information to all of the other Drill nodes so they
>> have
>> > > identical storage plugin instances registered. You do not need to
>> restart
>> > > any of the Drillbits when you add or update a storage plugin
>> instance."
>> > >
>> > > Does that mean if I were to have the following on one node:
>> > > [root@mufeed-drill1 ~]# cat
>> > >
>> > >
>> >
>> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
>> > > {
>> > >   "storage": {
>> > >
>> > >     dfs: {
>> > >       type: "file",
>> > >       connection: "maprfs:///"
>> > >     },
>> > >
>> > >     cp: {
>> > >       type: "file",
>> > >       connection: "classpath:///"
>> > >     },
>> > >
>> > >     cp1: {
>> > >       type: "file",
>> > >       connection: "classpath:///"
>> > >     },
>> > >       }
>> > > }
>> > >
>> > > The same content would get replicated to other nodes, on a file
>> content
>> > > level? If yes, I do not see that happening. If not, please clarify.
>> > >
>> > > ---
>> > > Mufeed Usman
>> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> |
>> My
>> > > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
>> > > <http://mufeed.livejournal.com>
>> > >
>> >
>>
>
>

--20cf303bf88e277e9f04fcd84198--