drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-4591) Extend config system with site, node property files
Date Thu, 07 Apr 2016 15:55:25 GMT
Paul Rogers created DRILL-4591:

             Summary: Extend config system with site, node property files
                 Key: DRILL-4591
                 URL: https://issues.apache.org/jira/browse/DRILL-4591
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: Paul Rogers
            Priority: Minor

Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh
file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME
per node, and thus they copy these two files per-node.

The result is that the two files act as both the overall "site" configuration (for all nodes)
and the "per-node" configuration for that one node.

The approach works, but is awkward. Ideally, provide the option to have two files: one for
the site, one for the node.

drill-override.conf, site-env.sh: For the site
drill-node.conf, drill-env.sh: For the particular node

The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically
deploy the site-wide files. In that case, the site files cannot also act as per-node files.

Note that, above, site-env.sh is new. The existing file is supposed to be per-node, so the
new file is site-wide.

By contrast, the drill-node.conf per-node configuration is new as the existing file is supposed
to be site-wide.

The site-wide site-env.sh might, say, point to custom locations for log, conf, pid or other
file locations, or set custom JVM options.

Similarly, the per-node drill-node.conf might customize the per-node memory or other resources
if one node has a different amount of physical memory or other resources.

The existing files are identified with the DRILL_CONF_DIR environment variable. Add a new
DRILL_SITE_CONF_DIR variable to point to the site-wide files.

When loading, per-node files are loaded after the site-wide files (so that per-node values
override site-wide values.)

As described, the change should be fully backward compatible. The change will also work with
a YARN solution that "localizes" (copies to the local node) the site-wide files. 

This message was sent by Atlassian JIRA

View raw message