drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paul-rogers <...@git.apache.org>
Subject [GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Date Sun, 10 Sep 2017 02:53:49 GMT
Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/938#discussion_r137939122
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
    @@ -92,18 +92,20 @@
     
       // Hash Aggregate Options
     
    -  String HASHAGG_NUM_PARTITIONS = "drill.exec.hashagg.num_partitions";
       String HASHAGG_NUM_PARTITIONS_KEY = "exec.hashagg.num_partitions";
       LongValidator HASHAGG_NUM_PARTITIONS_VALIDATOR = new RangeLongValidator(HASHAGG_NUM_PARTITIONS_KEY,
1, 128); // 1 means - no spilling
    -  String HASHAGG_MAX_MEMORY = "drill.exec.hashagg.mem_limit";
       String HASHAGG_MAX_MEMORY_KEY = "exec.hashagg.mem_limit";
       LongValidator HASHAGG_MAX_MEMORY_VALIDATOR = new RangeLongValidator(HASHAGG_MAX_MEMORY_KEY,
0, Integer.MAX_VALUE);
       // min batches is used for tuning (each partition needs so many batches when planning
the number of partitions,
       // or reserve this number when calculating whether the remaining available memory is
too small and requires a spill.)
       // Low value may OOM (e.g., when incoming rows become wider), higher values use fewer
partitions but are safer
    -  String HASHAGG_MIN_BATCHES_PER_PARTITION = "drill.exec.hashagg.min_batches_per_partition";
    -  String HASHAGG_MIN_BATCHES_PER_PARTITION_KEY = "drill.exec.hashagg.min_batches_per_partition";
    -  LongValidator HASHAGG_MIN_BATCHES_PER_PARTITION_VALIDATOR = new RangeLongValidator(HASHAGG_MIN_BATCHES_PER_PARTITION_KEY,
2, 5);
    +  String HASHAGG_MIN_BATCHES_PER_PARTITION_KEY = "exec.hashagg.min_batches_per_partition";
    +  LongValidator HASHAGG_MIN_BATCHES_PER_PARTITION_VALIDATOR = new RangeLongValidator(HASHAGG_MIN_BATCHES_PER_PARTITION_KEY,
1, 5);
    +  // Can be turns off mainly for testing. Memory prediction is used to decide on when
to spill to disk; with this option off,
    --- End diff --
    
    turns --> turned


---

Mime
View raw message