Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Fri, 6 Oct 2017 03:25:03 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.13093958.1502411248000.286074.1507260303038@Atlassian.JIRA>
In-Reply-To: <JIRA.13093958.1502411248000@Atlassian.JIRA>
References: <JIRA.13093958.1502411248000@Atlassian.JIRA> <JIRA.13093958.1502411248979@jira-lw-us.apache.org>
Subject: [jira] [Commented] (DRILL-5716) Queue-based memory assignment for
 buffering operators
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 06 Oct 2017 03:25:15 -0000


    [ https://issues.apache.org/jira/browse/DRILL-5716?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1619=
4082#comment-16194082 ]=20

ASF GitHub Bot commented on DRILL-5716:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/928#discussion_r143082463
 =20
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/forem=
an/rm/DistributedQueryQueue.java ---
    @@ -0,0 +1,342 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or imp=
lied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.work.foreman.rm;
    +
    +import java.util.concurrent.TimeUnit;
    +
    +import javax.xml.bind.annotation.XmlRootElement;
    +
    +import org.apache.drill.exec.ExecConstants;
    +import org.apache.drill.exec.coord.ClusterCoordinator;
    +import org.apache.drill.exec.coord.DistributedSemaphore;
    +import org.apache.drill.exec.coord.DistributedSemaphore.DistributedLea=
se;
    +import org.apache.drill.exec.proto.UserBitShared.QueryId;
    +import org.apache.drill.exec.proto.helper.QueryIdHelper;
    +import org.apache.drill.exec.server.DrillbitContext;
    +import org.apache.drill.exec.server.options.SystemOptionManager;
    +
    +/**
    + * Distributed query queue which uses a Zookeeper distributed semaphor=
e to
    + * control queuing across the cluster. The distributed queue is actual=
ly two
    + * queues: one for "small" queries, another for "large" queries. Query=
 size is
    + * determined by the Planner's estimate of query cost.
    + * <p>
    + * This queue is configured using system options:
    + * <dl>
    + * <dt><tt>exec.queue.enable</tt>
    + * <dt>
    + * <dd>Set to true to enable the distributed queue.</dd>
    + * <dt><tt>exec.queue.large</tt>
    + * <dt>
    + * <dd>The maximum number of large queries to admit. Additional
    + * queries wait in the queue.</dd>
    + * <dt><tt>exec.queue.small</tt>
    + * <dt>
    + * <dd>The maximum number of small queries to admit. Additional
    + * queries wait in the queue.</dd>
    + * <dt><tt>exec.queue.threshold</tt>
    + * <dt>
    + * <dd>The cost threshold. Queries below this size are small, at
    + * or above this size are large..</dd>
    + * <dt><tt>exec.queue.timeout_millis</tt>
    + * <dt>
    + * <dd>The maximum time (in milliseconds) a query will wait in the
    + * queue before failing.</dd>
    + * </dl>
    + * <p>
    + * The above values are refreshed every five seconds. This aids perfor=
mance
    + * a bit in systems with very high query arrival rates.
    + */
    +
    +public class DistributedQueryQueue implements QueryQueue {
    +  private static final org.slf4j.Logger logger =3D org.slf4j.LoggerFac=
tory.getLogger(DistributedQueryQueue.class);
    +
    +  private class DistributedQueueLease implements QueueLease {
    +    private final QueryId queryId;
    +    private DistributedLease lease;
    +    private final String queueName;
    +
    +    /**
    +     * Memory allocated to the query. Though all queries in the queue =
use
    +     * the same memory allocation rules, those rules can change at any=
 time
    +     * as the user changes system options. This value captures the val=
ue
    +     * calculated at the time that this lease was granted.
    +     */
    +    private long queryMemory;
    +
    +    public DistributedQueueLease(QueryId queryId, String queueName,
    +                    DistributedLease lease, long queryMemory) {
    +      this.queryId =3D queryId;
    +      this.queueName =3D queueName;
    +      this.lease =3D lease;
    +      this.queryMemory =3D queryMemory;
    +    }
    +
    +    @Override
    +    public String toString() {
    +      return String.format("Lease for %s queue to query %s",
    +          queueName, QueryIdHelper.getQueryId(queryId));
    +    }
    +
    +    @Override
    +    public long queryMemoryPerNode() { return queryMemory; }
    +
    +    @Override
    +    public void release() {
    +      DistributedQueryQueue.this.release(this);
    +    }
    +
    +    @Override
    +    public String queueName() { return queueName; }
    +  }
    +
    +  /**
    +   * Exposes a snapshot of internal state information for use in statu=
s
    +   * reporting, such as in the UI.
    +   */
    +
    +  @XmlRootElement
    +  public static class ZKQueueInfo {
    +    public final int smallQueueSize;
    +    public final int largeQueueSize;
    +    public final double queueThreshold;
    +    public final long memoryPerNode;
    +    public final long memoryPerSmallQuery;
    +    public final long memoryPerLargeQuery;
    +
    +    public ZKQueueInfo(DistributedQueryQueue queue) {
    +      smallQueueSize =3D queue.configSet.smallQueueSize;
    +      largeQueueSize =3D queue.configSet.largeQueueSize;
    +      queueThreshold =3D queue.configSet.queueThreshold;
    +      memoryPerNode =3D queue.memoryPerNode;
    +      memoryPerSmallQuery =3D queue.memoryPerSmallQuery;
    +      memoryPerLargeQuery =3D queue.memoryPerLargeQuery;
    +    }
    +  }
    +
    +  public interface StatusAdapter {
    --- End diff --
   =20
    Seems overkill for a one-line interface... I've tended to keep small, "=
helper" classes and interfaces in the file with the primary class, just to =
keep file clutter under control.


> Queue-based memory assignment for buffering operators
> -----------------------------------------------------
>
>                 Key: DRILL-5716
>                 URL: https://issues.apache.org/jira/browse/DRILL-5716
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.11.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>              Labels: doc-impacting
>             Fix For: 1.12.0
>
>
> Apache Drill already has a queueing feature based on ZK semaphores. We di=
d a bit of testing to  show that the feature does, in fact work. We propose=
 to enhance the feature with some light revisions to make work with the "ma=
naged" external sort and the newly-added spilling feature for the hash agg =
operator. The key requirement is to build on what we have for now; we may w=
ant to tackle a larger project to create a more complete solution later.
> Existing functionality:
> * Two ZK-based queues called the =E2=80=9Csmall=E2=80=9D and =E2=80=9Clar=
ge=E2=80=9D query queues.
> * A threshold, call it T, given as a query cost, to determine the queue i=
nto which a query will go.
> * Admit levels for the two queues: call them Qs and Ql.
> Basically, when a query comes in:
> * Plan the query as usual.
> * Obtain the final query cost from the planner, call this C.
> * If C<T, the query goes into the small queue, else it goes into the larg=
e queue.
> * Suppose the small queue. Ask ZK if the query can run.
> * ZK checks if Qs queries are already running. If so, the query waits, el=
se the query runs.
> The proposed changes include:
> * Refactor the code to provide a queueing API that supports a variety of =
queuing mechanisms.
> * Provide three: the null queue (default), an in-process queue (for testi=
ng) and the ZK queues.
> * Modify the query profile web UI to show two new bits of information abo=
ut queues:
> - The queue to which the query was sent.
> - The total planning cost.
> * Modify the query profile web UI to show two memory assignment numbers:
> - Total memory allocated to the query
> - Memory per sort or hash-add operator
> Then, add to the queue mechanism the ability to do memory assignment:
> * Provide a weight, W: every small query gets 1 unit, every large query g=
ets W units.
> * Use the queue admit levels to determine total units: U =3D Qs + W * Ql.
> * Obtain total direct memory from the system. M.
> * Subtract a reserve percent R for overhead.
> * Do the math to get the memory per query for each query:
> * For the small queue: (M - R) / U
> * For the large queue: (M - R) / U * W
> * Use this memory amount as the =E2=80=9Cmemory per query=E2=80=9D number=
 in the existing sort/hash-agg memory assignment (instead of the fixed 2 GB=
.)
> The result will be a nice incremental addition to what we already have, a=
nd should make it a bit easier people to actually use the feature (because =
they can see the planning numbers and see the queues used, allowing them to=
 effectively tune the system.)
> The API used for the above features also allow third parties to add on a =
more robust admission control feature as needed, perhaps tying into an exis=
ting queueing mechanism of their choice.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)