drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aditya Kishore (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-855) Improve work assignment parallelization
Date Thu, 29 May 2014 06:16:02 GMT

     [ https://issues.apache.org/jira/browse/DRILL-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Aditya Kishore updated DRILL-855:

    Attachment: DRILL-855-Improve-work-assignment-parallelization.patch

> Improve work assignment parallelization
> ---------------------------------------
>                 Key: DRILL-855
>                 URL: https://issues.apache.org/jira/browse/DRILL-855
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Aditya Kishore
>            Assignee: Aditya Kishore
>         Attachments: DRILL-855-Improve-work-assignment-parallelization.patch
> Currently if only a small number of endpoints have data affinity, all the work units
get assigned to this small subset leaving other endpoints in the cluster underutilized.
> It will lead to poor query performance since the level of parallelization is reduced
below what is otherwise possible.
> This JIRA intend to modify the assignment to do a slightly more equitable distribution
of work fragments among all available nodes while slightly favoring the nodes with data affinity.
> This \*favor\* is controlled by the setting {{"drill.exec.work.affinity.factor"}} (defaults
to 1.2f).
> As an example, let's say that there are 2 endpoints of which only one is with affinity
and 10 work fragments. In the current scheme of things, all of these work fragments will get
assigned to the single node with affinity. With the modified scheme and with default affinity
factor of 1.2, it will get 20% more assignment that it would get under equal distribution
(i.e. 6:4 instead of 5:5).

This message was sent by Atlassian JIRA

View raw message