drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-1170) YARN support for Drill
Date Thu, 26 Oct 2017 16:48:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220752#comment-16220752
] 

ASF GitHub Bot commented on DRILL-1170:
---------------------------------------

GitHub user paul-rogers opened a pull request:

    https://github.com/apache/drill/pull/1011

    Drill 1170: Drill-on-YARN

    Provides Drill integration with YARN. Runs Drill as a long-running task under YARN. Monitors
the Drill cluster, restarting failed Drillbits. Provides a command-line UI to start, stop
and resize the cluster. Provides a web-based UI to monitor the cluster.
    
    The Drill-on-YARN (DoY) code has been in use by commercial users for over a year, since
Drill 1.8 and has proven quite stable. Usage has been on MapR's version of YARN, we seek feedback
from users of the Apache and other versions of YARN.
    
    See [DRILL-1170](https://issues.apache.org/jira/browse/DRILL-1170) for design information.
See the included `README.md`` for internals information and `USAGE.md` for a detailed user
guide.
    
    This is a large PR; it will take time to review. The key goal at this moment is to allow
interested users to download the PR, build DoY, and try it out in their environments. The
DoY code is mostly independent of Drill itself. The DoY code can be used to launch any version
of Drill since 1.8. See the usage guide for information.
    
    It has been suggested that the code move to the `contrib` directory. That change will
be made. But, since the code works successfully in its current location; we'll leave it their
for now to ensure users are successful if they choose to try it.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/paul-rogers/drill DRILL-1170

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1011
    
----
commit e26012bc56cad3bf2819dff3bbdf70664de34955
Author: Paul Rogers <progers@maprtech.com>
Date:   2017-10-26T07:24:00Z

    DRILL-1170: YARN integration for Drill
    
    This commit includes documentation files.

commit 3a6ffe78d9fe0e9a5beacd100e2e0ee6b40c7f34
Author: Paul Rogers <progers@maprtech.com>
Date:   2017-10-26T07:25:34Z

    Client app

commit 509410c9710fc1ff23a4a13f51320a4c153f9328
Author: Paul Rogers <progers@maprtech.com>
Date:   2017-10-26T07:26:50Z

    Files common to several modules

commit 36b8d323118077115ef1097ba8b23f6fc4a5390a
Author: Paul Rogers <progers@maprtech.com>
Date:   2017-10-26T07:42:17Z

    Application master

commit 7fbc387634bb1acaf7807b02348b40364c81d282
Author: Paul Rogers <progers@maprtech.com>
Date:   2017-10-26T07:44:33Z

    App Master web UI

commit 21fb93792290625b719899a4742573b8c3d4a7ce
Author: Paul Rogers <progers@maprtech.com>
Date:   2017-10-26T07:45:44Z

    Distribution and project files

commit 567d36787b9ada60dd2141077e629158c53fc0c4
Author: Paul Rogers <progers@maprtech.com>
Date:   2017-10-26T07:47:54Z

    Test files

----


> YARN support for Drill
> ----------------------
>
>                 Key: DRILL-1170
>                 URL: https://issues.apache.org/jira/browse/DRILL-1170
>             Project: Apache Drill
>          Issue Type: New Feature
>            Reporter: Neeraja
>            Assignee: Paul Rogers
>             Fix For: Future
>
>         Attachments: Drill-on-YARNDesignOverview.pdf, Drill-on-YARNUserGuide.pdf
>
>
> This is a tracking item to make Drill work with YARN.
> Below are few requirements/needs to consider.
> - Drill should run as an YARN based application, side by side with other YARN enabled
applications (on same nodes or different nodes). Both memory and CPU resources of Drill should
be controlled in this mechanism.
> - As an YARN enabled application, Drill resource consumption should be adaptive to the
load on the cluster. For ex: When there is no load on the Drill , Drill should consume no
resources on the cluster.  As the load on Drill increases, resources permitting, usage should
grow proportionally.
> - Low latency is a key requirement for Apache Drill along with support for multiple users
(concurrency in 100s-1000s). This should be supported when run as YARN application as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message