drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paul-rogers <...@git.apache.org>
Subject [GitHub] drill pull request #960: DRILL-5815: Option to set query memory as percent o...
Date Mon, 16 Oct 2017 18:19:53 GMT
Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/960#discussion_r144926648
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/util/MemoryAllocationUtilities.java
---
    @@ -56,30 +59,90 @@ public static void setupBufferedOpsMemoryAllocations(final PhysicalPlan
plan, fi
         }
     
         // if there are any sorts, compute the maximum allocation, and set it on them
    -    if (bufferedOpList.size() > 0) {
    -      final OptionManager optionManager = queryContext.getOptions();
    -      double cpu_load_average = optionManager.getOption(ExecConstants.CPU_LOAD_AVERAGE);
    -      final long maxWidth = optionManager.getOption(ExecConstants.MAX_WIDTH_PER_NODE);
    -      final long maxWidthPerNode = ExecConstants.MAX_WIDTH_PER_NODE.computeMaxWidth(cpu_load_average,maxWidth);
    -      long maxAllocPerNode = Math.min(DrillConfig.getMaxDirectMemory(),
    -          queryContext.getConfig().getLong(RootAllocatorFactory.TOP_LEVEL_MAX_ALLOC));
    -      maxAllocPerNode = Math.min(maxAllocPerNode,
    -          optionManager.getOption(ExecConstants.MAX_QUERY_MEMORY_PER_NODE_KEY).num_val);
    -      final long maxOperatorAlloc = maxAllocPerNode / (bufferedOpList.size() * maxWidthPerNode);
    -      logger.debug("Max buffered operator alloc: {}", maxOperatorAlloc);
    -
    -      // User configurable option to allow forcing minimum memory.
    -      // Ensure that the buffered ops receive the minimum memory needed to make progress.
    -      // Without this, the math might work out to allocate too little memory.
    -      final long opMinMem = queryContext.getOptions().getOption(ExecConstants.MIN_MEMORY_PER_BUFFERED_OP_KEY).num_val;
    -
    -      for(final PhysicalOperator op : bufferedOpList) {
    -
    -        long alloc = Math.max(maxOperatorAlloc, op.getInitialAllocation());
    -        alloc = Math.max(alloc, opMinMem);
    -        op.setMaxAllocation(alloc);
    -      }
    -    }
         plan.getProperties().hasResourcePlan = true;
    +    if (bufferedOpList.isEmpty()) {
    +      return;
    +    }
    +
    +    // Setup options, etc.
    +
    +    final OptionManager optionManager = queryContext.getOptions();
    +    final long directMemory = DrillConfig.getMaxDirectMemory();
    +
    +    // Compute per-node, per-query memory.
    +
    +    final long maxAllocPerNode = computeQueryMemory(queryContext.getConfig(), optionManager,
directMemory);
    +    logger.debug("Memory per query per node: {}", maxAllocPerNode);
    +
    +    // Now divide up the memory by slices and operators.
    +
    +    final long opMinMem = computeOperatorMemory(optionManager, maxAllocPerNode, bufferedOpList.size());
    --- End diff --
    
    MEP? Do you mean Drill 1.13?
    
    The dynamic memory idea needs thought. In fact, this is exactly what we do today: each
operator grabs the memory it needs; and sends messages downstream if it is memory-constrained.
The reason we are doing so much work is that the existing technique is very, very hard to
get right.
    
    We are moving toward fixed budgets. Once that works, we can set a fixed budget per fragment
and "slosh" memory between operators in the same fragment based on an empirical study of memory
use.
    
    It will be far, far harder to slosh memory between fragments as a fragment lifecycle,
with buffering operators, is caret-shaped: ramp up to a peak, then ramp down. Coordination
is needed to know, if a fragment is using only 100 MB of 1 GB, that this is because the fragment
just started, and is ramping up (to full 1 GB use), vs. the operator is ramping down and won't
be using additional memory.
    
    Since all of this is a big project, the present change tries to do the best we can with
the situation we currently find ourselves in.


---

Mime
View raw message