drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-4893) Define in Drill-override.conf a common DFS root directory
Date Wed, 14 Sep 2016 20:39:20 GMT
Paul Rogers created DRILL-4893:

             Summary: Define in Drill-override.conf a common DFS root directory
                 Key: DRILL-4893
                 URL: https://issues.apache.org/jira/browse/DRILL-4893
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: Paul Rogers
            Priority: Minor

Drill-on-YARN and Dynamic UDFs are two Drill features that write files to the distributed
file system (DFS) such as HDFS. These files are configuration-style files (Drill software
for DoY, UDF jars for Dynamic UDFs.) Likely other Drill features will also use DFS for storage,
perhaps for logs, metadata, etc.

At present, each feature defines its own DFS storage path. An administrator who wants to define
a DFS location must know of, and modify, each of these locations.

The proposal here is to define a single Drill root directory property. Then, define all other
DFS locations relative to the root. This allows the admin to relocate all Drill files by changing
a single property. (Of course, this change should be made at install time, not after Drill
has written files, unless the files themselves are moved.)

For example (using made-up property names):

drill.dfs: “/user/drill”
drill.yarn.dfs: “${drill.dfs}/doy”
drill.exec.udf.directory.base: “${drill.dfs}/udf”
drill.exec.udf.directory.staging: “${drill.exec.udf.directory.base}/staging”

This message was sent by Atlassian JIRA

View raw message