drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bridg...@apache.org
Subject drill git commit: edits to planner.memory.max_query_memory_per_node description
Date Thu, 17 Aug 2017 22:47:38 GMT
Repository: drill
Updated Branches:
  refs/heads/gh-pages 505bc1425 -> bfe570561


edits to planner.memory.max_query_memory_per_node description


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/bfe57056
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/bfe57056
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/bfe57056

Branch: refs/heads/gh-pages
Commit: bfe570561e271696af1581604c5111608a0b915d
Parents: 505bc14
Author: Bridget Bevens <bbevens@maprtech.com>
Authored: Thu Aug 17 15:46:31 2017 -0700
Committer: Bridget Bevens <bbevens@maprtech.com>
Committed: Thu Aug 17 15:46:31 2017 -0700

----------------------------------------------------------------------
 .../010-configuration-options-introduction.md                    | 4 ++--
 ...050-sort-based-and-hash-based-memory-constrained-operators.md | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/drill/blob/bfe57056/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
b/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
index e153c9a..65566f5 100644
--- a/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
+++ b/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
@@ -1,6 +1,6 @@
 ---
 title: "Configuration Options Introduction"
-date: 2017-08-17 21:46:31 UTC
+date: 2017-08-17 22:46:40 UTC
 parent: "Configuration Options"
 ---
 
@@ -68,7 +68,7 @@ The sys.options table lists ptions that you can set at the system or session
lev
 | planner.memory.enable_memory_estimation           | FALSE                             
               | Toggles   the state of memory estimation and re-planning of the query. When
enabled,   Drill conservatively estimates memory requirements and typically excludes   these
operators from the plan and negatively impacts performance.                              
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
  
                                                                                         
                                                                                         
                                         |
 | planner.memory.hash_agg_table_factor              | 1.1                               
               | A   heuristic value for influencing the size of the hash aggregation table.
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
     
                                                                                         
                                                                                         
                                         |
 | planner.memory.hash_join_table_factor             | 1.1                               
               | A   heuristic value for influencing the size of the hash aggregation table.
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
     
                                                                                         
                                                                                         
                                         |
-| planner.memory.max_query_memory_per_node          | 2147483648   bytes                
               | Sets   the maximum amount of direct memory allocated to the Sort and Hash
Aggregate   operators during each query on a node. This memory is split between   operators.
If a query plan contains multiple Sort and/or Hash Aggregate   operators, the memory is divided
between them. The default setting is very   conservative.                                
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                        
                                                                                         
                                                                                         
                                         |
+| planner.memory.max_query_memory_per_node          | 2147483648   bytes                
               | Sets   the maximum amount of direct memory allocated to the Sort and Hash
Aggregate   operators during each query on a node. This memory is split between   operators.
If a query plan contains multiple Sort and/or Hash Aggregate   operators, the memory is divided
between them. The default setting is too small for queries on large data sets and should be
increased.                                                                               
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                      
                                                                                         
                                                                                         
                                                                                      |
 | planner.memory.non_blocking_operators_memory      | 64                                
               | Extra   query memory per node for non-blocking operators. This option is
currently   used only for memory estimation. Range: 0-2048 MB                            
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
        
                                                                                         
                                                                                         
                                         |
 | planner.memory_limit                              | 268435456   bytes                 
               | Defines   the maximum amount of direct memory allocated to a query for planning.
When   multiple queries run concurrently, each query is allocated the amount of   memory set
by this parameter.Increase the value of this parameter and rerun   the query if partition
pruning failed due to insufficient memory.                                               
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                       
                                                                                         
                                                                                         
                                         |
 | planner.nestedloopjoin_factor                     | 100                               
               | A   heuristic value for influencing the nested loop join.               
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
        
                                                                                         
                                                                                         
                                         |

http://git-wip-us.apache.org/repos/asf/drill/blob/bfe57056/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
----------------------------------------------------------------------
diff --git a/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
b/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
index 1c9de18..bb62602 100644
--- a/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
+++ b/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
@@ -1,6 +1,6 @@
 ---
 title: "Sort-Based and Hash-Based Memory-Constrained Operators"
-date: 2017-08-17 21:20:22 UTC
+date: 2017-08-17 22:46:43 UTC
 parent: "Query Plans and Tuning"
 --- 
 
@@ -9,7 +9,7 @@ Drill uses hash-based and sort-based operators depending on the query characteri
 When planning a query with sort- and hash-based operations, Drill evaluates the available
memory multiplied by a configurable reduction constant (for parallelization purposes) and
then limits the operations to the maximum of this amount of memory. Drill spills data to disk
if the sort and hash aggregate operations cannot be performed in memory. Alternatively, you
can disable large hash operations if they do not fit in memory on your system. When disabled,
Drill creates alternative plans. You can also modify the minimum hash table size, increasing
the size for very large aggregations or joins when you have large amounts of memory for Drill
to use. If you have large data sets, you can increase the hash table size to improve performance.

 
 ##Memory Options
-The `planner.memory.max_query_memory_per_node` option sets the maximum amount of direct memory
allocated to the Sort and Hash Aggregate operators during each query on a node. The default
limit is 2147483648 bytes (2GB), which is quite conservative. This memory is split between
operators. If a query plan contains multiple Sort and/or Hash Aggregate operators, the memory
is divided between them.
+The `planner.memory.max_query_memory_per_node` option sets the maximum amount of direct memory
allocated to the Sort and Hash Aggregate operators during each query on a node. The default
limit is set to 2147483648 bytes (2GB), which is too small for queries on large data sets
and should be increased. This memory is split between operators. If a query plan contains
multiple Sort and/or Hash Aggregate operators, the memory is divided between them.
 
 When a query is parallelized, the number of operators is multiplied, which reduces the amount
of memory given to each instance of the Sort and Hash Aggregate operators during a query.
If you encounter memory issues when running queries with Sort and Hash Aggregate operators,
calculate the memory requirements for your queries and the amount of available memory on each
node. Based on the information, increase the value of the `planner.memory.max_query_memory_per_node`
option using the ALTER SYSTEM|SESSION SET command, as shown:  
 


Mime
View raw message