drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arina Ielchiieva (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (DRILL-4052) Allow variables in drill-override.conf
Date Thu, 10 Mar 2016 18:11:41 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arina Ielchiieva reassigned DRILL-4052:

    Assignee: Arina Ielchiieva

> Allow variables in drill-override.conf
> --------------------------------------
>                 Key: DRILL-4052
>                 URL: https://issues.apache.org/jira/browse/DRILL-4052
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Tools, Build & Test
>    Affects Versions: 1.1.0
>         Environment: All
>            Reporter: John Omernik
>            Assignee: Arina Ielchiieva
>              Labels: features
>             Fix For: Future
> Basic variables, starting with "$HOSTNAME" if that simplified things would be a huge
help to managing clusters of drill bits, especially in multi-tenant environments.  
> (Note: Perhaps being able to use all ENV variables through a $ENV:ENVVARNAME would address
the issue completely, giving ultimate flexibility). 
> The use case/story I have is a drill cluster where I am running the individual drill
bits in Marathon on Mesos. Since I am using MapR-FS (although this could work without MapR
through Marathon downloading the config at run time) I point all drill bits to one config
location.   This allows me to make a change in one place, making for easy changes and configuration
> For the drill-env.sh, since that is shell script, I found I could easily put in items
like specifying log locations that allowed me to use one shared log location for all drillbits,
and then specifying that each logfile had the hostname of the drillbit, so I could keep all
logs together and thus troubleshoot much easier.  
> When it came to drill-override.conf, I wanted to specify spill directory locations. Specifically,
I wanted to use local volumes in MapR-FS.  It allows me to use a clustered filesystem, but
have that data only exit on the node that the drill bit is running on.  So, Scheme: maprfs
works, but what to specify for the location? I didn't want to specify the same location, /tmp/spill
as then it would be shared by drillbits, and not guaranteed to be local to the drill bit (giving
up on performance).  I wanted to create a volume that could be mounted /tmp/$hostname/drillspill
  I could create the volume in drill-env.sh, but I could not specify that in the drill-override
without creating a separate override file for each node. 
> Given scaling and multiple users/clusters, I really want to avoid that situation (a config
for each drill bit), hence creating this JIRA for the ability to use variables in the drill-override.
Happy to discuss other use cases as well. 

This message was sent by Atlassian JIRA

View raw message