drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-1170) YARN support for Drill
Date Tue, 22 Mar 2016 23:57:25 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207566#comment-15207566
] 

Paul Rogers commented on DRILL-1170:
------------------------------------

A brief "starter set" of requirements:

* Configuration file to gather the cluster configuration (memory, cores, number of nodes and
so on.)
* Launcher to start/stop Drill within YARN
* Drill-specific Application Master (AM)
* AM requests YARN Node Manager (AM) to launch drill-bits.
* Use YARN localization feature to depoy Drill files to each node.
* Add nodes (drill-bits) to a running Drill cluster
* Remove nodes from a running Drill cluster (see DRILL-2656)
* Detect and restart failed drill-bits
* Status/statistics about the cluster as a whole (number of active nodes, number of restarts,
etc.)
* Allow existing users to run "unmanaged" Drill clusters (YARN is optional)
* Possibly allow multiple "Drill clusters" (independent clusters of drill bits) on the same
YARN-managed physical cluster.


> YARN support for Drill
> ----------------------
>
>                 Key: DRILL-1170
>                 URL: https://issues.apache.org/jira/browse/DRILL-1170
>             Project: Apache Drill
>          Issue Type: New Feature
>            Reporter: Neeraja
>            Assignee: Paul Rogers
>             Fix For: Future
>
>
> This is a tracking item to make Drill work with YARN.
> Below are few requirements/needs to consider.
> - Drill should run as an YARN based application, side by side with other YARN enabled
applications (on same nodes or different nodes). Both memory and CPU resources of Drill should
be controlled in this mechanism.
> - As an YARN enabled application, Drill resource consumption should be adaptive to the
load on the cluster. For ex: When there is no load on the Drill , Drill should consume no
resources on the cluster.  As the load on Drill increases, resources permitting, usage should
grow proportionally.
> - Low latency is a key requirement for Apache Drill along with support for multiple users
(concurrency in 100s-1000s). This should be supported when run as YARN application as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message