drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5716) Queue-based memory assignment for buffering operators
Date Fri, 01 Sep 2017 21:51:01 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151207#comment-16151207

ASF GitHub Bot commented on DRILL-5716:

Github user paul-rogers commented on a diff in the pull request:

    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/DefaultResourceManager.java
    @@ -0,0 +1,122 @@
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.work.foreman.rm;
    +import org.apache.drill.common.config.DrillConfig;
    +import org.apache.drill.exec.ops.QueryContext;
    +import org.apache.drill.exec.physical.PhysicalPlan;
    +import org.apache.drill.exec.server.BootStrapContext;
    +import org.apache.drill.exec.util.MemoryAllocationUtilities;
    +import org.apache.drill.exec.work.QueryWorkUnit;
    +import org.apache.drill.exec.work.foreman.Foreman;
    + * Represents a default resource manager for clusters that do not provide query
    + * queues. Without queues to provide a hard limit on the query admission rate,
    + * the number of active queries must be estimated and the resulting resource
    + * allocations will be rough estimates.
    + */
    +public class DefaultResourceManager implements ResourceManager {
    +  public static class DefaultResourceAllocator implements QueryResourceAllocator {
    +    private QueryContext queryContext;
    +    protected DefaultResourceAllocator(QueryContext queryContext) {
    +      this.queryContext = queryContext;
    +    }
    +    @Override
    +    public void visitAbstractPlan(PhysicalPlan plan) {
    +      boolean replanMemory = ! plan.getProperties().hasResourcePlan;
    +      if (! replanMemory || plan == null) {
    +        return;
    +      }
    +      MemoryAllocationUtilities.setupBufferedOpsMemoryAllocations(plan, queryContext);
    +    }
    +    @Override
    +    public void visitPhysicalPlan(QueryWorkUnit work) {
    +    }
    +  }
    +  public static class DefaultQueryResourceManager extends DefaultResourceAllocator implements
QueryResourceManager {
    +    @SuppressWarnings("unused")
    +    private final DefaultResourceManager rm;
    +    public DefaultQueryResourceManager(final DefaultResourceManager rm, final Foreman
foreman) {
    +      super(foreman.getQueryContext());
    +      this.rm = rm;
    +    }
    +    @Override
    +    public void setCost(double cost) {
    +      // Nothing to do. The EXECUTION option in Foreman calls this,
    +      // but does not do the work to plan sort memory. Is EXECUTION
    +      // even used?
    --- End diff --
    Exactly, that is the point. There is a code path in the Foreman that figures out the cost,
then does nothing with it. There seem to be multiple ways that we work with plans, and I struggled
to understand them all.
    Anyway, since the comment is not useful to anyone but me, I removed it.

> Queue-based memory assignment for buffering operators
> -----------------------------------------------------
>                 Key: DRILL-5716
>                 URL: https://issues.apache.org/jira/browse/DRILL-5716
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.12.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
> Apache Drill already has a queueing feature based on ZK semaphores. We did a bit of testing
to  show that the feature does, in fact work. We propose to enhance the feature with some
light revisions to make work with the "managed" external sort and the newly-added spilling
feature for the hash agg operator. The key requirement is to build on what we have for now;
we may want to tackle a larger project to create a more complete solution later.
> Existing functionality:
> * Two ZK-based queues called the “small” and “large” query queues.
> * A threshold, call it T, given as a query cost, to determine the queue into which a
query will go.
> * Admit levels for the two queues: call them Qs and Ql.
> Basically, when a query comes in:
> * Plan the query as usual.
> * Obtain the final query cost from the planner, call this C.
> * If C<T, the query goes into the small queue, else it goes into the large queue.
> * Suppose the small queue. Ask ZK if the query can run.
> * ZK checks if Qs queries are already running. If so, the query waits, else the query
> The proposed changes include:
> * Refactor the code to provide a queueing API that supports a variety of queuing mechanisms.
> * Provide three: the null queue (default), an in-process queue (for testing) and the
ZK queues.
> * Modify the query profile web UI to show two new bits of information about queues:
> - The queue to which the query was sent.
> - The total planning cost.
> * Modify the query profile web UI to show two memory assignment numbers:
> - Total memory allocated to the query
> - Memory per sort or hash-add operator
> Then, add to the queue mechanism the ability to do memory assignment:
> * Provide a weight, W: every small query gets 1 unit, every large query gets W units.
> * Use the queue admit levels to determine total units: U = Qs + W * Ql.
> * Obtain total direct memory from the system. M.
> * Subtract a reserve percent R for overhead.
> * Do the math to get the memory per query for each query:
> * For the small queue: (M - R) / U
> * For the large queue: (M - R) / U * W
> * Use this memory amount as the “memory per query” number in the existing sort/hash-agg
memory assignment (instead of the fixed 2 GB.)
> The result will be a nice incremental addition to what we already have, and should make
it a bit easier people to actually use the feature (because they can see the planning numbers
and see the queues used, allowing them to effectively tune the system.)
> The API used for the above features also allow third parties to add on a more robust
admission control feature as needed, perhaps tying into an existing queueing mechanism of
their choice.

This message was sent by Atlassian JIRA

View raw message