Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Fri, 11 Aug 2017 00:28:00 +0000 (UTC)
From: "Paul Rogers (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.13093958.1502411248000.5932.1502411280136@Atlassian.JIRA>
In-Reply-To: <JIRA.13093958.1502411248000@Atlassian.JIRA>
References: <JIRA.13093958.1502411248000@Atlassian.JIRA> <JIRA.13093958.1502411248979@jira-lw-us.apache.org>
Subject: [jira] [Created] (DRILL-5716) Queue-based memory assignment for
 buffering operators
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 11 Aug 2017 00:28:06 -0000

Paul Rogers created DRILL-5716:
----------------------------------

             Summary: Queue-based memory assignment for buffering operators
                 Key: DRILL-5716
                 URL: https://issues.apache.org/jira/browse/DRILL-5716
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.12.0
            Reporter: Paul Rogers
            Assignee: Paul Rogers


Apache Drill already has a queueing feature based on ZK semaphores. We did =
a bit of testing to  show that the feature does, in fact work. We propose t=
o enhance the feature with some light revisions to make work with the "mana=
ged" external sort and the newly-added spilling feature for the hash agg op=
erator. The key requirement is to build on what we have for now; we may wan=
t to tackle a larger project to create a more complete solution later.

Existing functionality:

* Two ZK-based queues called the =E2=80=9Csmall=E2=80=9D and =E2=80=9Clarge=
=E2=80=9D query queues.
* A threshold, call it T, given as a query cost, to determine the queue int=
o which a query will go.
* Admit levels for the two queues: call them Qs and Ql.

Basically, when a query comes in:

* Plan the query as usual.
* Obtain the final query cost from the planner, call this C.
* If C<T, the query goes into the small queue, else it goes into the large =
queue.
* Suppose the small queue. Ask ZK if the query can run.
* ZK checks if Qs queries are already running. If so, the query waits, else=
 the query runs.

The proposed changes include:

* Refactor the code to provide a queueing API that supports a variety of qu=
euing mechanisms.
* Provide three: the null queue (default), an in-process queue (for testing=
) and the ZK queues.
* Modify the query profile web UI to show two new bits of information about=
 queues:
- The queue to which the query was sent.
- The total planning cost.
* Modify the query profile web UI to show two memory assignment numbers:
- Total memory allocated to the query
- Memory per sort or hash-add operator

Then, add to the queue mechanism the ability to do memory assignment:

* Provide a weight, W: every small query gets 1 unit, every large query get=
s W units.
* Use the queue admit levels to determine total units: U =3D Qs + W * Ql.
* Obtain total direct memory from the system. M.
* Subtract a reserve percent R for overhead.
* Do the math to get the memory per query for each query:
* For the small queue: (M - R) / U
* For the large queue: (M - R) / U * W
* Use this memory amount as the =E2=80=9Cmemory per query=E2=80=9D number i=
n the existing sort/hash-agg memory assignment (instead of the fixed 2 GB.)

The result will be a nice incremental addition to what we already have, and=
 should make it a bit easier people to actually use the feature (because th=
ey can see the planning numbers and see the queues used, allowing them to e=
ffectively tune the system.)

The API used for the above features also allow third parties to add on a mo=
re robust admission control feature as needed, perhaps tying into an existi=
ng queueing mechanism of their choice.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)