drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-1170) YARN support for Drill
Date Tue, 29 Mar 2016 18:00:28 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216504#comment-15216504

Paul Rogers commented on DRILL-1170:

We have considered Slider. Several factors nudged us in the direction of writing an AM directly
on YARN:

1. Slider has much documentation, but it is incomplete and out-of-date in important places.
2. We could make up for the documenation by reading the source code. However, Slider is composed
of a large amount of Python code. Our team are mostly Java developers. If we have to learn
a bunch of code, we might as well learn YARN directly.
3. Drill needs certain features that Slider does not (yet) provide, such as monitoring ZooKeeper
to track Drill-bit health, perhaps offering a connection proxy, etc.
4. Slider is a general-purpose tool with many cool features. As it turns out, many are not
needed for Drill. This means that Slider introduces a bit of unnecessary complexity for Drill
5. Slider adds its own level of configuration files on top of those that we'd need for Drill.
Not a big issue, but it is just additional complexity for Drill admins to learn and manage.

In balance, we like where Slider is going. Those Drill users who want to roll-their-own YARN
integration should certainly give Slider a try as a short-term solution. This is particularly
true for shops that already use Slider for other apps.

On balance, however, Drill has a number of specialized needs that would seem to justify the
cost of a custom AM. We will, of course, continue to revisit the issue as analysis proceeds.

> YARN support for Drill
> ----------------------
>                 Key: DRILL-1170
>                 URL: https://issues.apache.org/jira/browse/DRILL-1170
>             Project: Apache Drill
>          Issue Type: New Feature
>            Reporter: Neeraja
>            Assignee: Paul Rogers
>             Fix For: Future
> This is a tracking item to make Drill work with YARN.
> Below are few requirements/needs to consider.
> - Drill should run as an YARN based application, side by side with other YARN enabled
applications (on same nodes or different nodes). Both memory and CPU resources of Drill should
be controlled in this mechanism.
> - As an YARN enabled application, Drill resource consumption should be adaptive to the
load on the cluster. For ex: When there is no load on the Drill , Drill should consume no
resources on the cluster.  As the load on Drill increases, resources permitting, usage should
grow proportionally.
> - Low latency is a key requirement for Apache Drill along with support for multiple users
(concurrency in 100s-1000s). This should be supported when run as YARN application as well.

This message was sent by Atlassian JIRA

View raw message