Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Wed, 30 Mar 2016 17:31:25 +0000 (UTC)
From: "Jacques Nadeau (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.12953786.1459024327000.90656.1459359085543@Atlassian.JIRA>
In-Reply-To: <JIRA.12953786.1459024327000@Atlassian.JIRA>
References: <JIRA.12953786.1459024327000@Atlassian.JIRA>
 <JIRA.12953786.1459024327186@arcas>
Subject: [jira] [Commented] (DRILL-4543) Advertise Drill-bit ports, status,
 capabilities in ZooKeeper
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/DRILL-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218403#comment-15218403 ] 

Jacques Nadeau commented on DRILL-4543:
---------------------------------------

Important clarification here, the current file is per drillbit. Conceptually you need to think of separating:

bootstrap options
runtime options

bootstrap options are per node and are needed to start the engine. They are defined in drill-override.conf (and the supporting defaults files)
system options are per cluster and can be set at runtime. They are defined in the Drill OptionManager hierarchy

You override ordering statement is a bit confusing (since items lower in the list are higher priority)

It seems extreme to try to provide all options as ENV variables. If there are a few that we should add as stadard available, great. However, it doesn't make sense to reimplement the entire HOCON system to also support ENV variables (especially since you can't even encapsulate appending to a list for example using env variables). 


> Advertise Drill-bit ports, status, capabilities in ZooKeeper
> ------------------------------------------------------------
>
>                 Key: DRILL-4543
>                 URL: https://issues.apache.org/jira/browse/DRILL-4543
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components:  Server
>            Reporter: Paul Rogers
>             Fix For: 2.0.0
>
>
> Today Drill uses ZooKeeper (ZK) to advertise the existence of a Drill-bit, providing the host name/IP Address of the Drill-bit and the ports used, encoded in Protobuf format. All other information (status, CPUs, memory) are assumed to be the same across all Drill-bits in the cluster as specified in the Drill config file. (Amended to reflect 1.6 behavior.)
> Moving forward, as Drill becomes more sophisticated, Drill should advertise the specifics of each Drill-bit so that one Drill bit can differ from another.
> For example, when running on YARN, we need a way for Drill to gracefully shut down. Advertising a status of Ready or Unavailable will help. Ready is the normal state. Unavailable means the Drill-bit will finish in-flight queries, but won't accept new ones. (The actual status is a separate enhancement.)
> In a YARN cluster, Drill should take advantage of machines with more memory, but live with machines with less. (Perhaps some are newer, some are older or more heavily loaded.) Drill should use ZK to identify its available memory and CPUs so that the planner can use them. (Use of the info is a separate enhancement.)
> There may be times when two drill bits run on a single machine. If so, they must use separate ports. So, each Drill-bit should advertise its ports in ZK.
> For backward compatibility, the information is optional; if not present, the receiver should assume the information defaults to that in the config file.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)