drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bridg...@apache.org
Subject drill-site git commit: Additional doc updates related to hash agg spill to disk for 1.11
Date Thu, 17 Aug 2017 21:26:33 GMT
Repository: drill-site
Updated Branches:
  refs/heads/asf-site 53b591ef7 -> ecf68552c


Additional doc updates related to hash agg spill to disk for 1.11


Project: http://git-wip-us.apache.org/repos/asf/drill-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill-site/commit/ecf68552
Tree: http://git-wip-us.apache.org/repos/asf/drill-site/tree/ecf68552
Diff: http://git-wip-us.apache.org/repos/asf/drill-site/diff/ecf68552

Branch: refs/heads/asf-site
Commit: ecf68552c90c572648007de3908ffa0ef1e6eea1
Parents: 53b591e
Author: Bridget Bevens <bbevens@maprtech.com>
Authored: Thu Aug 17 14:26:17 2017 -0700
Committer: Bridget Bevens <bbevens@maprtech.com>
Committed: Thu Aug 17 14:26:17 2017 -0700

----------------------------------------------------------------------
 docs/configuring-drill-memory/index.html        | 27 +++++-----
 .../index.html                                  | 50 +++++++++---------
 docs/start-up-options/index.html                | 53 ++++++++++----------
 feed.xml                                        |  4 +-
 4 files changed, 68 insertions(+), 66 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/drill-site/blob/ecf68552/docs/configuring-drill-memory/index.html
----------------------------------------------------------------------
diff --git a/docs/configuring-drill-memory/index.html b/docs/configuring-drill-memory/index.html
index ad23cf3..acf24cc 100644
--- a/docs/configuring-drill-memory/index.html
+++ b/docs/configuring-drill-memory/index.html
@@ -1126,30 +1126,31 @@
 
     </div>
 
-     Nov 1, 2016
+     Aug 17, 2017
 
     <link href="/css/docpage.css" rel="stylesheet" type="text/css">
 
     <div class="int_text" align="left">
       
-        <p>You can configure the amount of direct memory allocated to a Drillbit for
query processing in any Drill cluster, multitenant or not. The default memory for a Drillbit
is 8G, but Drill prefers 16G or more depending on the workload. The total amount of direct
memory that a Drillbit allocates to query operations cannot exceed the limit set.</p>
+        <p>You can configure the amount of direct memory allocated to a Drillbit for
query processing in any Drill cluster, multitenant or not. The default memory for a drillbit
is 8G, but Drill prefers 16G or more depending on the workload. The total amount of direct
memory that a drillbit allocates to query operations cannot exceed the limit set.</p>
 
-<p>Drill uses Java direct memory and performs well when executing
-operations in memory instead of storing the operations on disk. Drill does not
-write to disk unless absolutely necessary, unlike MapReduce where everything
-is written to disk during each phase of a job.</p>
+<p>Drill uses Java direct memory and performs well when executing operations in memory
instead of storing the operations on disk. Drill does not write to disk unless absolutely
necessary, unlike MapReduce where everything is written to disk during each phase of a job.</p>
 
 <p>The JVM’s heap memory does not limit the amount of direct memory available in
-a Drillbit. The on-heap memory for Drill is typically set at 4-8G (default is 4), which should
+a drillbit. The on-heap memory for Drill is typically set at 4-8G (default is 4), which should
 suffice because Drill avoids having data sit in heap memory.</p>
 
-<p>As of Drill 1.5, Drill uses a new allocator that improves an operator’s use of
direct memory and tracks the memory use more accurately. Due to this change, the sort operator
(in queries that ran successfully in previous releases) may not have enough memory, resulting
in a failed query and out of memory error instead of spilling to disk.</p>
+<p>As of Drill 1.5, Drill uses a new allocator that improves an operator’s use of
direct memory and tracks the memory use more accurately. Due to this change, the sort operator
(in queries that ran successfully in previous releases) may not have enough memory, resulting
in a failed query and out of memory error instead of spilling to disk.     </p>
 
-<p>The <a href="/docs/configuration-options-introduction/#system-options"><code>planner.memory.max_query_memory_per_node</code></a>
system option value sets the maximum amount of direct memory allocated to the sort operator
in each query on a node. If a query plan contains multiple sort operators, they all share
this memory. If you encounter memory issues when running queries with sort operators, increase
the value of this option. If you continue to encounter memory issues after increasing this
value, you can also reduce the value of the <a href="/docs/configuration-options-introduction/"><code>planner.width.max_per_node</code></a>
option to reduce the level of parallelism per node. However, this may increase the amount
of time required for a query to complete.  </p>
+<h2 id="drillbit-memory">Drillbit Memory</h2>
 
-<h2 id="modifying-drillbit-memory">Modifying Drillbit Memory</h2>
+<p>The value set for the <a href="/docs/configuration-options-introduction/#system-options"><code>planner.memory.max_query_memory_per_node</code></a>
system option sets the maximum amount of direct memory allocated to the Sort and Hash Aggreate
operators in each query on a node. If a query plan contains multiple Sort and/or Hash Aggregate
operators, they all share this memory. If you encounter memory issues when running queries
with Sort and/or Hash Aggregate operators, increase the value of this option. See <a href="https://drill.apache.org/docs/sort-based-and-hash-based-memory-constrained-operators/">Sort-Based
and Hash-Based Memory Constrained Operators</a> for more information.  </p>
 
-<p>You can modify memory for each Drillbit node in your cluster. To modify the memory
for a Drillbit, set the DRILL_MAX_DIRECT_MEMORY variable in the Drillbit startup script, <code>drill-env.sh</code>,
located in <code>&lt;drill_installation_directory&gt;/conf</code>, as
follows:</p>
+<p>If you continue to encounter memory issues after increasing this value, you can
also reduce the value of the <a href="/docs/configuration-options-introduction/"><code>planner.width.max_per_node</code></a>
option to reduce the level of parallelism per node. However, this may increase the amount
of time required for a query to complete. </p>
+
+<h3 id="modifying-drillbit-memory">Modifying Drillbit Memory</h3>
+
+<p>You can modify memory for each drillbit node in your cluster. To modify the memory
for a drillbit, set the DRILL_MAX_DIRECT_MEMORY variable in the drillbit startup script, <code>drill-env.sh</code>,
located in <code>&lt;drill_installation_directory&gt;/conf</code>, as
follows:</p>
 <div class="highlight"><pre><code class="language-text" data-lang="text">export
DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-&quot;&lt;value&gt;&quot;}
 </code></pre></div>
 <div class="admonition note">
@@ -1157,9 +1158,9 @@ suffice because Drill avoids having data sit in heap memory.</p>
   <p class="last">If DRILL_MAX_DIRECT_MEMORY is not set, the limit depends on the amount
of available system memory.  </p>
 </div>
 
-<p>After you edit <code>&lt;drill_installation_directory&gt;/conf/drill-env.sh</code>,
<a href="/docs/starting-drill-in-distributed-mode">restart the Drillbit</a> on
the node.</p>
+<p>After you edit <code>&lt;drill_installation_directory&gt;/conf/drill-env.sh</code>,
<a href="/docs/starting-drill-in-distributed-mode">restart the drillbit</a> on
the node.</p>
 
-<h2 id="about-the-drillbit-startup-script">About the Drillbit startup script</h2>
+<h2 id="about-the-drillbit-startup-script">About the Drillbit Startup Script</h2>
 
 <p>The <code>drill-env.sh</code> file contains the following options:</p>
 <div class="highlight"><pre><code class="language-text" data-lang="text">#export
DRILL_HEAP=${DRILL_HEAP:-&quot;4G”}  

http://git-wip-us.apache.org/repos/asf/drill-site/blob/ecf68552/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
----------------------------------------------------------------------
diff --git a/docs/sort-based-and-hash-based-memory-constrained-operators/index.html b/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
index 84ae607..fd10d3b 100644
--- a/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
+++ b/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
@@ -1134,25 +1134,25 @@
 
     <div class="int_text" align="left">
       
-        <p>Drill uses hash-based and sort-based operators depending on the query characteristics.
Hash aggregation and hash join are hash-based operations. Streaming aggregation and merge
join are sort-based operations. Both hash-based and sort-based operations consume memory,
however the hash aggregate and hash join operators are the fastest and most memory intensive
operators. </p>
+        <p>Drill uses hash-based and sort-based operators depending on the query characteristics.
Hash Aggregate and Hash Join are hash-based operators. Sort, Streaming Aggregate, and Merge
Join are sort-based operators. Both hash-based and sort-based operations consume memory, however
the Hash Aggregate and Hash Join operators are the fastest and most memory intensive operators.
</p>
 
-<p>When planning a query with sort- and hash-based operators, Drill evaluates the available
memory multiplied by a configurable reduction constant (for parallelization purposes) and
then limits the operations to the maximum of this amount of memory. Drill spills data to disk
if the sort and hash aggregate operations cannot be performed in memory. Alternatively, you
can disable large hash operations if they do not fit in memory on your system. When disabled,
Drill creates alternative plans. You can also modify the minimum hash table size, increasing
the size for very large aggregations or joins when you have large amounts of memory for Drill
to use. If you have large data sets, you can increase the hash table size to improve performance.
</p>
+<p>When planning a query with sort- and hash-based operations, Drill evaluates the
available memory multiplied by a configurable reduction constant (for parallelization purposes)
and then limits the operations to the maximum of this amount of memory. Drill spills data
to disk if the sort and hash aggregate operations cannot be performed in memory. Alternatively,
you can disable large hash operations if they do not fit in memory on your system. When disabled,
Drill creates alternative plans. You can also modify the minimum hash table size, increasing
the size for very large aggregations or joins when you have large amounts of memory for Drill
to use. If you have large data sets, you can increase the hash table size to improve performance.
</p>
 
 <h2 id="memory-options">Memory Options</h2>
 
-<p>The <code>planner.memory.max_query_memory_per_node</code> option sets
the maximum amount of direct memory allocated to the sort and hash aggregate operators during
each query on a node. The default limit is 2147483648 bytes (2GB), which is quite conservative.
This memory is split between operators. If a query plan contains multiple sort and/or hash
aggregate operators, the memory is divided between them.</p>
+<p>The <code>planner.memory.max_query_memory_per_node</code> option sets
the maximum amount of direct memory allocated to the Sort and Hash Aggregate operators during
each query on a node. The default limit is 2147483648 bytes (2GB), which is quite conservative.
This memory is split between operators. If a query plan contains multiple Sort and/or Hash
Aggregate operators, the memory is divided between them.</p>
 
-<p>When a query is parallelized, the number of operators is multiplied, which reduces
the amount of memory given to each instance of the sort and hash aggregate operators during
a query. If you encounter memory issues when running queries with sort and hash aggregate
operators, calculate the memory requirements for your queries and the amount of available
memory on each node. Based on the information, increase the value for the <code>planner.memory.max_query_memory_per_node</code>
option using the ALTER SYSTEM|SESSION SET command, as shown:  </p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">ALTER
SYSTEM|SESSION SET `planner.memory.max_query_memory_per_node` = 8147483648  
+<p>When a query is parallelized, the number of operators is multiplied, which reduces
the amount of memory given to each instance of the Sort and Hash Aggregate operators during
a query. If you encounter memory issues when running queries with Sort and Hash Aggregate
operators, calculate the memory requirements for your queries and the amount of available
memory on each node. Based on the information, increase the value of the <code>planner.memory.max_query_memory_per_node</code>
option using the ALTER SYSTEM|SESSION SET command, as shown:  </p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">ALTER
SYSTEM|SESSION SET `planner.memory.max_query_memory_per_node` = &lt;new_value&gt;
 
 </code></pre></div>
-<p>The <code>planner.memory.enable_memory_estimation</code> option toggles
the state of memory estimation and re-planning of the query. When enabled, Drill conservatively
estimates memory requirements and typically excludes memory-constrained operators from the
query plan, which can negatively impact performance. The default setting is false. If you
want Drill to use very conservative memory estimates, use the ALTER SYSTEM|SESSION SET command
to change the setting, as shown:  </p>
+<p>The <code>planner.memory.enable_memory_estimation</code> option toggles
the state of memory estimation and re-planning of a query. When enabled, Drill conservatively
estimates memory requirements and typically excludes memory-constrained operators from the
query plan, which can negatively impact performance. The default setting is false. If you
want Drill to use very conservative memory estimates, use the ALTER SYSTEM|SESSION SET command
to change the setting, as shown:  </p>
 <div class="highlight"><pre><code class="language-text" data-lang="text">ALTER
SYSTEM|SESSION SET `planner.memory.enable_memory_estimation` = true  
 </code></pre></div>
 <h2 id="spill-to-disk">Spill to Disk</h2>
 
-<p>The &quot;Spill to Disk&quot; feature prevents queries that use memory-intensive
sort and hash aggregate operations from failing with out-of-memory errors. Drill automatically
writes excess data to a temporary directory on disk when queries with sort or hash aggregate
operations exceed the set memory limit on a Drill node. When the operators finish processing
the in-memory data, Drill reads the spilled data back from disk, and the operators finish
processing the data. When the operations complete, Drill removes the data from disk.  </p>
+<p>Spilling data to disk prevents queries that use memory-intensive Sort and Hash Aggregate
operations from failing with out-of-memory errors. Drill automatically writes excess data
to a temporary directory on disk when queries with Sort or Hash Aggregate operations exceed
the set memory limit on a Drill node. When the operators finish processing the in-memory data,
Drill reads the spilled data back from disk, and the operators finish processing the data.
When the operations complete, Drill removes the data from disk.  </p>
 
-<p>Spilling to disk enables queries to run uninterrupted while Drill performs the spill
operations in the background. However, there can be performance impact due to the time required
to spill data and then read the data back from disk.  </p>
+<p>Spilling data to disk enables queries to run uninterrupted while Drill performs
the spill operations in the background. However, there can be performance impact due to the
time required to spill data and then read the data back from disk.  </p>
 
 <div class="admonition note">
   <p class="first admonition-title">Note</p>
@@ -1173,7 +1173,7 @@
 
 <h3 id="spill-to-disk-configuration-options">Spill to Disk Configuration Options</h3>
 
-<p>The spill to disk options reside in the drill-override.conf file on each Drill node.
An administrator or someone familiar with storage and disks should manage these settings.</p>
+<p>The options related to spilling reside in the drill-override.conf file on each Drill
node. An administrator or someone familiar with storage and disks should manage these settings.</p>
 
 <div class="admonition note">
   <p class="first admonition-title">Note</p>
@@ -1184,37 +1184,37 @@
 
 <ul>
 <li><p><strong>drill.exe.spill.fs</strong><br>
-Introduced in Drill 1.11. The default file system on the local machine into which the sort
and hash aggregate operators spill data. This is the recommended option to use for spilling.
You can configure this option so that data spills into a distributed file system, such as
hdfs. For example, &quot;hdfs:///&quot;. The default setting is &quot;file:///&quot;.
 </p></li>
+Introduced in Drill 1.11. The default file system on the local machine into which the Sort
and Hash Aggregate operators spill data. This is the recommended option to use for spilling.
You can configure this option so that data spills into a distributed file system, such as
hdfs. For example, &quot;hdfs:///&quot;. The default setting is &quot;file:///&quot;.
 </p></li>
 <li><p><strong>drill.exec.spill.directories</strong><br>
-Introduced in Drill 1.11. The list of directories into which the sort and hash aggregate
operators spill data. The list must be an array with directories separated by a comma, for
example [&quot;/fs1/drill/spill&quot; , &quot;/fs2/drill/spill&quot; , &quot;/fs3/drill/spill&quot;].
This is the recommended option for spilling to multiple directories. The default setting is
[&quot;/tmp/drill/spill&quot;].  </p></li>
+Introduced in Drill 1.11. The list of directories into which the Sort and Hash Aggregate
operators spill data. The list must be an array with directories separated by a comma, for
example [&quot;/fs1/drill/spill&quot; , &quot;/fs2/drill/spill&quot; , &quot;/fs3/drill/spill&quot;].
This is the recommended option for spilling to multiple directories. The default setting is
[&quot;/tmp/drill/spill&quot;].  </p></li>
 <li><p><strong>drill.exec.sort.external.spill.fs</strong><br>
-Overrides the default location into which the sort operator spills data. Instead of spilling
into the location set by the drill.exec.spill.fs option, the sort operators spill into the
location specified by this option.<br>
-<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you   use the drill.exec.spill.fs option to set the spill location instead. The default
setting is &quot;file:///&quot;.</p></li>
+Overrides the default location into which the Sort operator spills data. Instead of spilling
into the location set by the <code>drill.exec.spill.fs</code> option, the Sort
operators spill into the location specified by this option.<br>
+<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you use the <code>drill.exec.spill.fs</code> option to set the spill location
instead. The default setting is &quot;file:///&quot;.  </p></li>
 <li><p><strong>drill.exec.sort.external.spill.directories</strong><br>
-Overrides the location into which the sort operator spills data. Instead of spilling into
the location set by the drill.exec.spill.directories option, the sort operators spill into
the directories specified by this option. The list must be an array with directories separated
by a comma, for example [&quot;/fs1/drill/spill&quot; , &quot;/fs2/drill/spill&quot;
, &quot;/fs3/drill/spill&quot;].<br>
-<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you use the drill.exec.spill.directories option to set the spill location instead. The
default setting is [&quot;/tmp/drill/spill&quot;].  </p></li>
+Overrides the location into which the Sort operator spills data. Instead of spilling into
the location set by the <code>drill.exec.spill.directories</code> option, the
Sort operators spill into the directories specified by this option. The list must be an array
with directories separated by a comma, for example [&quot;/fs1/drill/spill&quot; ,
&quot;/fs2/drill/spill&quot; , &quot;/fs3/drill/spill&quot;].<br>
+<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you use the <code>drill.exec.spill.directories</code> option to set the spill
location instead. The default setting is [&quot;/tmp/drill/spill&quot;].  </p></li>
 <li><p><strong>drill.exec.hashagg.spill.fs</strong><br>
-Overrides the location into which the hash aggregate operator spills data. Instead of spilling
into the location set by the drill.exec.spill.fs option, the hash aggregate operator spills
into the location specified by this option. Setting this option to 1 disables spilling for
the hash aggregate operator.<br>
-<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you use the drill.exec.spill.fs option to set the spill location instead. The default
setting is &quot;file:///&quot;.  </p></li>
+Overrides the location into which the Hash Aggregate operator spills data. Instead of spilling
into the location set by the <code>drill.exec.spill.fs</code> option, the Hash
Aggregate operator spills into the location specified by this option. Setting this option
to 1 disables spilling for the Hash Aggregate operator.<br>
+<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you use the <code>drill.exec.spill.fs</code> option to set the spill location
instead. The default setting is &quot;file:///&quot;.  </p></li>
 <li><p><strong>drill.exec.hashagg.spill.directories</strong><br>
-Overrides the location into which the hash aggregate operator spills data. Instead of spilling
into the location set by the drill.exec.spill.directories option, the hash aggregate operator
spills to the directories specified by this option. The list must be an array with directories
separated by a comma, for example [&quot;/fs1/drill/spill&quot; , &quot;/fs2/drill/spill&quot;
, &quot;/fs3/drill/spill&quot;].<br>
-<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you use the drill.exec.spill. directories option to set the spill location instead. 
</p></li>
+Overrides the location into which the Hash Aggregate operator spills data. Instead of spilling
into the location set by the <code>drill.exec.spill.directories</code> option,
the Hash Aggregate operator spills to the directories specified by this option. The list must
be an array with directories separated by a comma, for example [&quot;/fs1/drill/spill&quot;
, &quot;/fs2/drill/spill&quot; , &quot;/fs3/drill/spill&quot;].<br>
+<strong>Note:</strong> As of Drill 1.11, this option is supported for backward
compatibility, however in future releases, this option will be deprecated. It is highly recommended
that you use the <code>drill.exec.spill.directories option</code> to set the spill
location instead.  </p></li>
 </ul>
 
-<h2 id="hash-based-operator-settings">Hash-Based Operator Settings</h2>
+<h2 id="hash-based-operator-configuration-settings">Hash-Based Operator Configuration
Settings</h2>
 
-<p>Use the ALTER SYSTEM|SESSION SET commands with the options below to disable the
hash aggregate and hash join operators, modify the hash table size, disable memory estimation,
or set the estimated maximum amount of memory for a query. Typically, you set the options
at the session level unless you want the setting to persist across all sessions.</p>
+<p>Use the ALTER SYSTEM|SESSION SET commands with the options below to disable the
Hash Aggregate and Hash Join operators, modify the hash table size, or disable memory estimation.
Typically, you set the options at the session level unless you want the setting to persist
across all sessions.</p>
 
 <p>The following options control the hash-based operators:</p>
 
 <ul>
 <li><p><strong>planner.enable_hashagg</strong><br>
-Enables or disables hash aggregation; otherwise, Drill does a sort-based aggregation. This
option is enabled by default.   The default setting is true, which is recommended.</p></li>
+Enables or disables hash aggregation; otherwise, Drill does a sort-based aggregation. This
option is enabled by default. The default, and recommended, setting is true. 
+The Hash Aggregate operator uses an uncontrolled amount of memory, up to 10 GB, after which
the operator runs out of memory. As of Drill 1.11, the Hash Aggregate operator can write to
disk. </p></li>
 <li><p><strong>planner.enable_hashjoin</strong><br>
-Enables or disables the memory hungry hash join. Drill assumes that a query will have adequate
memory to complete and tries to use the fastest operations possible to complete the planned
inner, left, right, or full outer joins using a hash table. Currently, this operator does
not write to disk. Disabling hash join allows Drill to manage arbitrarily large data in a
small memory footprint. This option is enabled by default. The default setting is true.</p></li>
+Enables or disables the memory hungry hash join. Drill assumes that a query will have adequate
memory to complete and tries to use the fastest operations possible to complete the planned
inner, left, right, or full outer joins using a hash table. The Hash Join operator uses an
uncontrolled amount of memory, up to 10 GB, after which the operator runs out of memory. Currently,
this operator does not write to disk. Disabling hash join allows Drill to manage arbitrarily
large data in a small memory footprint. This option is enabled by default. The default setting
is true.</p></li>
 <li><p><strong>exec.min_hash_table_size</strong><br>
-Starting size for hash tables. Increase this setting based on the memory available to improve
performance.<br>
-The default setting for this option is 65536. The setting can range from 0 to 1073741824.</p></li>
+Starting size for hash tables. Increase this setting based on the memory available to improve
performance. The default setting for this option is 65536. The setting can range from 0 to
1073741824.</p></li>
 <li><p><strong>exec.max_hash_table_size</strong><br>
 Ending size for hash tables. The default setting for this option is 1073741824. The setting
can range from 0 to 1073741824.</p></li>
 </ul>

http://git-wip-us.apache.org/repos/asf/drill-site/blob/ecf68552/docs/start-up-options/index.html
----------------------------------------------------------------------
diff --git a/docs/start-up-options/index.html b/docs/start-up-options/index.html
index 1efbd32..39f1e67 100644
--- a/docs/start-up-options/index.html
+++ b/docs/start-up-options/index.html
@@ -1128,18 +1128,18 @@
 
     </div>
 
-     Aug 8, 2017
+     Aug 17, 2017
 
     <link href="/css/docpage.css" rel="stylesheet" type="text/css">
 
     <div class="int_text" align="left">
       
-        <p>Drill’s start-up options reside in a <a href="https://github.com/typesafehub/config/blob/master/HOCON.md">HOCON</a>
configuration file format, which is
-a hybrid between a properties file and a JSON file. Drill start-up options
-consist of a group of files with a nested relationship. At the bottom of the file hierarchy
are the default files that Drill provides, starting with <code>drill-default.conf</code>.
The <code>drill-default.conf</code> file is overridden by one or more <code>drill-module.conf</code>
files that Drill’s internal modules provide. The <code>drill-module.conf</code>
files are overridden by the <code>drill-override.conf</code> file that you define.
   </p>
+        <p>The start-up options for Drill reside in a <a href="https://github.com/typesafehub/config/blob/master/HOCON.md">HOCON</a>
configuration file format, which is a hybrid between a properties file and a JSON file. Drill
start-up options consist of a group of files with a nested relationship. At the bottom of
the file hierarchy are the default files that Drill provides, starting with <code>drill-default.conf</code>.
</p>
 
-<p>You can provide overrides on each Drillbit using system properties of the form <code>-Dname=value</code>
passed on the command line: </p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">
  ./drillbit.sh start -Dname=value
+<p>The <code>drill-default.conf</code> file is overridden by one or more
<code>drill-module.conf</code> files that Drill’s internal modules provide.
The <code>drill-module.conf</code> files are overridden by the <code>drill-override.conf</code>
file that you define.    </p>
+
+<p>You can provide overrides on each drillbit using system properties of the form <code>-Dname=value</code>
passed on the command line: </p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">./drillbit.sh
start -Dname=value
 </code></pre></div>
 <p>You can see the following group of files throughout the source repository in
 Drill:</p>
@@ -1151,44 +1151,45 @@ contrib/storage-hive/hive-exec-shade/src/main/resources/drill-module.conf
 exec/java-exec/src/main/resources/drill-module.conf
 distribution/src/resources/drill-override.conf
 </code></pre></div>
-<p>These files are listed inside the associated JAR files in the Drill
-distribution tarball.</p>
+<p>These files are listed inside the associated JAR files in the Drill distribution
tarball.</p>
 
 <p>Each Drill module has a set of options that Drill incorporates. Drill’s
 modular design enables you to create new storage plugins, set new operators,
 or create UDFs. You can also include additional configuration options that you
-can override as necessary.</p>
+can override as needed.</p>
 
 <p>When you add a JAR file to Drill, you must include a <code>drill-module.conf</code>
file
 in the root directory of the JAR file that you add. The <code>drill-module.conf</code>
 file tells Drill to scan that JAR file or associated object and include it.</p>
 
-<h2 id="viewing-startup-options">Viewing Startup Options</h2>
+<h2 id="viewing-start-up-options">Viewing Start-Up Options</h2>
 
-<p>You can run the following query to see a list of Drill’s startup options:</p>
+<p>Run the following query to see a list of the available start-up options:</p>
 <div class="highlight"><pre><code class="language-text" data-lang="text">SELECT
* FROM sys.boot;
 </code></pre></div>
 <h2 id="configuring-start-up-options">Configuring Start-Up Options</h2>
 
-<p>You can configure start-up options for each Drillbit in <code>&lt;drill_home&gt;/conf/drill-override.conf</code>
.</p>
+<p>You can configure start-up options for each drillbit in <code>&lt;drill_home&gt;/conf/drill-override.conf</code>
.</p>
 
 <p>The summary of start-up options, also known as boot options, lists default values.
The following descriptions provide more detail on key options that are frequently reconfigured:</p>
 
 <ul>
-<li><strong>drill.exec.http.ssl_enabled</strong><br>
-Available in Drill 1.2. Enables or disables <a href="/docs/configuring-web-console-and-rest-api-security/#https-support">HTTPS
support</a>. Settings are TRUE and FALSE, respectively. The default is FALSE.<br></li>
-<li><strong>drill.exec.sys.store.provider.class</strong><br>
-Defines the persistent storage (PStore) provider. The <a href="/docs/persistent-configuration-storage">PStore</a>
holds configuration and profile data.<br></li>
-<li><strong>drill.exec.buffer.size</strong><br>
-Defines the amount of memory available, in terms of record batches, to hold data on the downstream
side of an operation. Drill pushes data downstream as quickly as possible to make data immediately
available. This requires Drill to use memory to hold the data pending operations. When data
on a downstream operation is required, that data is immediately available so Drill does not
have to go over the network to process it. Providing more memory to this option increases
the speed at which Drill completes a query.<br></li>
-<li><strong>drill.exec.sort.external.spill.directories</strong><br>
-Tells Drill which directory to use when spooling. Drill uses a spool and sort operation for
beyond memory operations. The sorting operation is designed to spool to a Hadoop file system.
The default Hadoop file system is a local file system in the <code>/tmp</code>
directory. Spooling performance (both writing and reading back from it) is constrained by
the file system.<br></li>
-<li><strong>drill.exec.zk.connect</strong><br>
-Provides Drill with the ZooKeeper quorum to use to connect to data sources. Change this setting
to point to the ZooKeeper quorum that you want Drill to use. You must configure this option
on each Drillbit node.<br></li>
-<li><strong>drill.exec.profiles.store.inmemory</strong><br>
-Available as of Drill 1.11. When set to TRUE, enables Drill to store query profiles in memory
instead of writing the query profiles to disk. When set to FALSE, Drill writes the profile
for each query to disk, which is either the local file system or a distributed file system,
such as HDFS. For sub-second queries, writing the query profile to disk is expensive due to
the interactions with the file system. Enable this option if you want Drill to store the profiles
of sub-second queries in memory instead of writing them to disk. When you enable this option,
Drill stores the profiles in memory for as long as the drillbit runs. When the drillbit restarts,
the profiles no longer exist. You can set the maximum number of most recent profiles to retain
in memory through the drill.exec.profiles.store.capacity option. Settings are TRUE and FALSE.
Default is FALSE.<br></li>
-<li><strong>drill.exec.profiles.store.capacity</strong><br>
-Available as of Drill 1.11. Sets the maximum number of most recent profiles to retain in
memory when the drill.exec.profiles.store.inmemory option is enabled. Default is 1000.<br></li>
+<li><p><strong>drill.exec.http.ssl_enabled</strong><br>
+Available in Drill 1.2. Enables or disables <a href="/docs/configuring-web-console-and-rest-api-security/#https-support">HTTPS
support</a>. Settings are TRUE and FALSE, respectively. The default is FALSE.  </p></li>
+<li><p><strong>drill.exec.sys.store.provider.class</strong><br>
+Defines the persistent storage (PStore) provider. The <a href="/docs/persistent-configuration-storage">PStore</a>
holds configuration and profile data.  </p></li>
+<li><p><strong>drill.exec.buffer.size</strong><br>
+Defines the amount of memory available, in terms of record batches, to hold data on the downstream
side of an operation. Drill pushes data downstream as quickly as possible to make data immediately
available. This requires Drill to use memory to hold the data pending operations. When data
on a downstream operation is required, that data is immediately available so Drill does not
have to go over the network to process it. Providing more memory to this option increases
the speed at which Drill completes a query.  </p></li>
+<li><p><strong>drill.exe.spill.fs</strong><br>
+Introduced in Drill 1.11. The default file system on the local machine into which the Sort
and Hash Aggregate operators spill data. This is the recommended option to use for spilling.
You can configure this option so that data spills into a distributed file system, such as
hdfs. For example, &quot;hdfs:///&quot;. The default setting is &quot;file:///&quot;.
See <a href="/docs/sort-based-and-hash-based-memory-constrained-operators/">Sort-Based
and Hash-Based Memory Constrained Operators</a> for more information.   </p></li>
+<li><p><strong>drill.exec.spill.directories</strong><br>
+Introduced in Drill 1.11. The list of directories into which the Sort and Hash Aggregate
operators spill data. The list must be an array with directories separated by a comma, for
example [&quot;/fs1/drill/spill&quot; , &quot;/fs2/drill/spill&quot; , &quot;/fs3/drill/spill&quot;].
This is the recommended option for spilling to multiple directories. The default setting is
[&quot;/tmp/drill/spill&quot;]. See <a href="/docs/sort-based-and-hash-based-memory-constrained-operators/">Sort-Based
and Hash-Based Memory Constrained Operators</a> for more information.  </p></li>
+<li><p><strong>drill.exec.zk.connect</strong><br>
+Provides Drill with the ZooKeeper quorum to use to connect to data sources. Change this setting
to point to the ZooKeeper quorum that you want Drill to use. You must configure this option
on each Drillbit node.  </p></li>
+<li><p><strong>drill.exec.profiles.store.inmemory</strong><br>
+Available as of Drill 1.11. When set to TRUE, enables Drill to store query profiles in memory
instead of writing the query profiles to disk. When set to FALSE, Drill writes the profile
for each query to disk, which is either the local file system or a distributed file system,
such as HDFS. For sub-second queries, writing the query profile to disk is expensive due to
the interactions with the file system. Enable this option if you want Drill to store the profiles
of sub-second queries in memory instead of writing them to disk. When you enable this option,
Drill stores the profiles in memory for as long as the drillbit runs. When the drillbit restarts,
the profiles no longer exist. You can set the maximum number of most recent profiles to retain
in memory through the <code>drill.exec.profiles.store.capacity</code> option.
Settings are TRUE and FALSE. Default is FALSE. See <a href="/docs/persistent-configuration-storage/">Persistent
Configuration Storage</a> for more information.  </p>
 </li>
+<li><p><strong>drill.exec.profiles.store.capacity</strong><br>
+Available as of Drill 1.11. Sets the maximum number of most recent profiles to retain in
memory when the <code>drill.exec.profiles.store.inmemory</code> option is enabled.
Default is 1000.  </p></li>
 </ul>
 
     

http://git-wip-us.apache.org/repos/asf/drill-site/blob/ecf68552/feed.xml
----------------------------------------------------------------------
diff --git a/feed.xml b/feed.xml
index b7bcc10..b0eecf5 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
 </description>
     <link>/</link>
     <atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
-    <pubDate>Thu, 17 Aug 2017 12:03:10 -0700</pubDate>
-    <lastBuildDate>Thu, 17 Aug 2017 12:03:10 -0700</lastBuildDate>
+    <pubDate>Thu, 17 Aug 2017 14:23:20 -0700</pubDate>
+    <lastBuildDate>Thu, 17 Aug 2017 14:23:20 -0700</lastBuildDate>
     <generator>Jekyll v2.5.2</generator>
     
       <item>


Mime
View raw message