drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-4591) Extend config system with distrib, site, node property files
Date Fri, 08 Apr 2016 23:19:25 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Paul Rogers updated DRILL-4591:
    Summary: Extend config system with distrib, site, node property files  (was: Extend config
system with site, node property files)

> Extend config system with distrib, site, node property files
> ------------------------------------------------------------
>                 Key: DRILL-4591
>                 URL: https://issues.apache.org/jira/browse/DRILL-4591
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Paul Rogers
>            Priority: Minor
> Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh
file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME
per node, and thus they copy these two files per-node.
> The result is that the two files act as both the overall "site" configuration (for all
nodes) and the "per-node" configuration for that one node.
> The approach works, but is awkward. Ideally, provide the option to have two files: one
for the site, one for the node.
> drill-override.conf, drill-env.sh: For the site
> drill-override.conf, drill-env.sh: For the particular node
> The site-wide site-env.sh might, say, point to custom locations for log, conf, pid or
other file locations, or set custom JVM options.
> Similarly, the per-node drill-node.conf might customize the per-node memory or other
resources if one node has a different amount of physical memory or other resources.
> The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically
deploy the site-wide files. In that case, the site files cannot also act as per-node files.
> Existing installs have only one each of drill-override.conf, drill-env.sh, copied to
each node and possibly modified. For compatibility, the existing files become the site-wide
files. When converting to a split-setup, copy the site-wide files as before. But, rather than
modifying the site-wide files per-node, create a new per-load location, create new per-node
files, and place the per-node modifications in those files. The per-node files need not change
on following upgrades, just copy the site-wide files and that's it.
> We assume that DRILL_CONF_DIR points to the site-wide files. It defaults to $DRILL_HOME/conf
or in /etc/drill/conf. To use node-specific files, the admin must set up a new, per-node location
and identify that location using a proposed DRILL_NODE_CONF_DIR variable.
> The config files should be optional: skip them if they do not exist (or, for node files,
skip them if DRILL_NODE_CONF_DIR is not set.)
> When loading, per-node files are loaded after the site-wide files (so that per-node values
override site-wide values.)
> The change should be fully backward compatible. The change will also work with a YARN
solution that "localizes" (copies to the local node) the site-wide files. 

This message was sent by Atlassian JIRA

View raw message