impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jbap...@apache.org
Subject [13/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.
Date Thu, 17 Nov 2016 23:11:51 GMT
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_stats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_stats.xml b/docs/topics/impala_perf_stats.xml
new file mode 100644
index 0000000..33d394d
--- /dev/null
+++ b/docs/topics/impala_perf_stats.xml
@@ -0,0 +1,1031 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="perf_stats">
+
+  <title>Table and Column Statistics</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala can do better optimization for complex or multi-table queries when it has access to statistics about
+      the volume of data and how the values are distributed. Impala uses this information to help parallelize and
+      distribute the work for a query. For example, optimizing join queries requires a way of determining if one
+      table is <q>bigger</q> than another, which is a function of the number of rows and the average row size
+      for each table. The following sections describe the categories of statistics Impala can work
+      with, and how to produce them and keep them up to date.
+    </p>
+
+    <note>
+      <p rev="1.2.2">
+        Originally, Impala relied on the Hive mechanism for collecting statistics, through the Hive <codeph>ANALYZE
+        TABLE</codeph> statement which initiates a MapReduce job. For better user-friendliness and reliability,
+        Impala implements its own <codeph>COMPUTE STATS</codeph> statement in Impala 1.2.2 and higher, along with the
+        <codeph>DROP STATS</codeph>, <codeph>SHOW TABLE STATS</codeph>, and <codeph>SHOW COLUMN STATS</codeph>
+        statements.
+      </p>
+    </note>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="perf_table_stats">
+
+    <title id="table_stats">Overview of Table Statistics</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+<!-- Hive background info: https://cwiki.apache.org/Hive/statsdev.html -->
+
+      <p>
+        The Impala query planner can make use of statistics about entire tables and partitions.
+        This information includes physical characteristics such as the number of rows, number of data files,
+        the total size of the data files, and the file format. For partitioned tables, the numbers
+        are calculated per partition, and as totals for the whole table.
+        This metadata is stored in the metastore database, and can be updated by either Impala or Hive.
+        If a number is not available, the value -1 is used as a placeholder.
+        Some numbers, such as number and total sizes of data files, are always kept up to date because
+        they can be calculated cheaply, as part of gathering HDFS block metadata.
+      </p>
+
+      <p>
+        The following example shows table stats for an unpartitioned Parquet table.
+        The values for the number and sizes of files are always available.
+        Initially, the number of rows is not known, because it requires a potentially expensive
+        scan through the entire table, and so that value is displayed as -1.
+        The <codeph>COMPUTE STATS</codeph> statement fills in any unknown table stats values.
+      </p>
+
+<codeblock>
+show table stats parquet_snappy;
++-------+--------+---------+--------------+-------------------+---------+-------------------+...
+| #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format  | Incremental stats |...
++-------+--------+---------+--------------+-------------------+---------+-------------------+...
+| -1    | 96     | 23.35GB | NOT CACHED   | NOT CACHED        | PARQUET | false             |...
++-------+--------+---------+--------------+-------------------+---------+-------------------+...
+
+compute stats parquet_snappy;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 6 column(s). |
++-----------------------------------------+
+
+
+show table stats parquet_snappy;
++------------+--------+---------+--------------+-------------------+---------+-------------------+...
+| #Rows      | #Files | Size    | Bytes Cached | Cache Replication | Format  | Incremental stats |...
++------------+--------+---------+--------------+-------------------+---------+-------------------+...
+| 1000000000 | 96     | 23.35GB | NOT CACHED   | NOT CACHED        | PARQUET | false             |...
++------------+--------+---------+--------------+-------------------+---------+-------------------+...
+</codeblock>
+
+      <p>
+        Impala performs some optimizations using this metadata on its own, and other optimizations by
+        using a combination of table and column statistics.
+      </p>
+
+      <p rev="1.2.1">
+        To check that table statistics are available for a table, and see the details of those statistics, use the
+        statement <codeph>SHOW TABLE STATS <varname>table_name</varname></codeph>. See
+        <xref href="impala_show.xml#show"/> for details.
+      </p>
+
+      <p>
+        If you use the Hive-based methods of gathering statistics, see
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/StatsDev" scope="external" format="html">the
+        Hive wiki</xref> for information about the required configuration on the Hive side. <ph rev="upstream">Cloudera</ph> recommends
+        using the Impala <codeph>COMPUTE STATS</codeph> statement to avoid potential configuration and scalability
+        issues with the statistics-gathering process.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+    </conbody>
+  </concept>
+
+  <concept id="perf_column_stats">
+
+    <title id="column_stats">Overview of Column Statistics</title>
+
+    <conbody>
+
+<!-- Cloudera+Hive background information: http://blog.cloudera.com/blog/2012/08/column-statistics-in-hive/ -->
+
+      <p>
+        The Impala query planner can make use of statistics about individual columns when that metadata is
+        available in the metastore database. This technique is most valuable for columns compared across tables in
+        <xref href="impala_perf_joins.xml#perf_joins">join queries</xref>, to help estimate how many rows the query
+        will retrieve from each table. <ph rev="2.0.0"> These statistics are also important for correlated
+        subqueries using the <codeph>EXISTS()</codeph> or <codeph>IN()</codeph> operators, which are processed
+        internally the same way as join queries.</ph>
+      </p>
+
+      <p>
+        The following example shows column stats for an unpartitioned Parquet table.
+        The values for the maximum and average sizes of some types are always available,
+        because those figures are constant for numeric and other fixed-size types.
+        Initially, the number of distinct values is not known, because it requires a potentially expensive
+        scan through the entire table, and so that value is displayed as -1.
+        The same applies to maximum and average sizes of variable-sized types, such as <codeph>STRING</codeph>.
+        The <codeph>COMPUTE STATS</codeph> statement fills in most unknown column stats values.
+        (It does not record the number of <codeph>NULL</codeph> values, because currently Impala
+        does not use that figure for query optimization.)
+      </p>
+
+<codeblock>
+show column stats parquet_snappy;
++-------------+----------+------------------+--------+----------+----------+
+| Column      | Type     | #Distinct Values | #Nulls | Max Size | Avg Size |
++-------------+----------+------------------+--------+----------+----------+
+| id          | BIGINT   | -1               | -1     | 8        | 8        |
+| val         | INT      | -1               | -1     | 4        | 4        |
+| zerofill    | STRING   | -1               | -1     | -1       | -1       |
+| name        | STRING   | -1               | -1     | -1       | -1       |
+| assertion   | BOOLEAN  | -1               | -1     | 1        | 1        |
+| location_id | SMALLINT | -1               | -1     | 2        | 2        |
++-------------+----------+------------------+--------+----------+----------+
+
+compute stats parquet_snappy;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 6 column(s). |
++-----------------------------------------+
+
+show column stats parquet_snappy;
++-------------+----------+------------------+--------+----------+-------------------+
+| Column      | Type     | #Distinct Values | #Nulls | Max Size | Avg Size          |
++-------------+----------+------------------+--------+----------+-------------------+
+| id          | BIGINT   | 183861280        | -1     | 8        | 8                 |
+| val         | INT      | 139017           | -1     | 4        | 4                 |
+| zerofill    | STRING   | 101761           | -1     | 6        | 6                 |
+| name        | STRING   | 145636240        | -1     | 22       | 13.00020027160645 |
+| assertion   | BOOLEAN  | 2                | -1     | 1        | 1                 |
+| location_id | SMALLINT | 339              | -1     | 2        | 2                 |
++-------------+----------+------------------+--------+----------+-------------------+
+</codeblock>
+
+      <note>
+        <p>
+          For column statistics to be effective in Impala, you also need to have table statistics for the
+          applicable tables, as described in <xref href="impala_perf_stats.xml#perf_table_stats"/>. When you use
+          the Impala <codeph>COMPUTE STATS</codeph> statement, both table and column statistics are automatically
+          gathered at the same time, for all columns in the table.
+        </p>
+        <p conref="../shared/impala_common.xml#common/decimal_no_stats"/>
+      </note>
+
+      <note conref="../shared/impala_common.xml#common/compute_stats_nulls"/>
+
+<!-- Hive-based instructions are considered obsolete since the introduction of the Impala COMPUTE STATS statement.
+      <p>
+        Add settings like the following to the <filepath>hive-site.xml</filepath>
+        configuration file, in the Hive configuration directory, on every node where you run
+        <codeph>ANALYZE TABLE</codeph> statements through the
+        <codeph>hive</codeph> shell. The
+        <codeph>hive.stats.ndv.error</codeph> setting represents the standard error when
+        estimating the number of distinct values for a column. The value of 5.0 is recommended as a tradeoff between the
+        accuracy of the gathered statistics and the resource usage of the stats-gathering process.
+      </p>
+
+<codeblock><![CDATA[<property>
+ <name>hive.stats.ndv.error</name>
+ <value>5.0</value>
+</property>]]></codeblock>
+
+      <p>
+        5.0 is a relatively low value that devotes substantial computational resources to the statistics-gathering
+        process. To reduce the resource usage, you could increase this value; to make the statistics even more precise,
+        you could lower it.
+      </p>
+
+      <p>
+        The syntax for gathering column statistics uses the <codeph>ANALYZE TABLE ...
+        COMPUTE STATISTICS</codeph> clause, with an additional <codeph>FOR
+        COLUMNS</codeph> clause. For partitioned tables, you can gather statistics for specific partitions by including
+        a clause <codeph>PARTITION
+        (<varname>col1=val1</varname>,<varname>col2=val2</varname>,
+        ...)</codeph>; but you cannot include the partitioning columns in the
+        <codeph>FOR COLUMNS</codeph> clause. Also, you cannot use fully qualified table
+        names, so issue a <codeph>USE</codeph> command first to switch to the
+        appropriate database. For example:
+      </p>
+
+<codeblock>USE <varname>database_name</varname>;
+ANALYZE TABLE <varname>table_name</varname> COMPUTE STATISTICS FOR COLUMNS <varname>column_list</varname>;
+ANALYZE TABLE <varname>table_name</varname> PARTITION (<varname>partition_specs</varname>) COMPUTE STATISTICS FOR COLUMNS <varname>column_list</varname>;</codeblock>
+-->
+
+      <p rev="1.2.1">
+        To check whether column statistics are available for a particular set of columns, use the <codeph>SHOW
+        COLUMN STATS <varname>table_name</varname></codeph> statement, or check the extended
+        <codeph>EXPLAIN</codeph> output for a query against that table that refers to those columns. See
+        <xref href="impala_show.xml#show"/> and <xref href="impala_explain.xml#explain"/> for details.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+    </conbody>
+  </concept>
+
+  <concept id="perf_stats_partitions">
+    <title id="stats_partitions">How Table and Column Statistics Work for Partitioned Tables</title>
+    <conbody>
+
+      <p>
+        When you use Impala for <q>big data</q>, you are highly likely to use partitioning
+        for your biggest tables, the ones representing data that can be logically divided
+        based on dates, geographic regions, or similar criteria. The table and column statistics
+        are especially useful for optimizing queries on such tables. For example, a query involving
+        one year might involve substantially more or less data than a query involving a different year,
+        or a range of several years. Each query might be optimized differently as a result.
+      </p>
+
+      <p>
+        The following examples show how table and column stats work with a partitioned table.
+        The table for this example is partitioned by year, month, and day.
+        For simplicity, the sample data consists of 5 partitions, all from the same year and month.
+        Table stats are collected independently for each partition. (In fact, the
+        <codeph>SHOW PARTITIONS</codeph> statement displays exactly the same information as
+        <codeph>SHOW TABLE STATS</codeph> for a partitioned table.) Column stats apply to
+        the entire table, not to individual partitions. Because the partition key column values
+        are represented as HDFS directories, their characteristics are typically known in advance,
+        even when the values for non-key columns are shown as -1.
+      </p>
+
+<codeblock>
+show partitions year_month_day;
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| year  | month | day | #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format  |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| 2013  | 12    | 1   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 2   | -1    | 1      | 2.53MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 3   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 4   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 5   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| Total |       |     | -1    | 5      | 12.58MB | 0B           |                   |         |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+
+show table stats year_month_day;
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| year  | month | day | #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format  |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| 2013  | 12    | 1   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 2   | -1    | 1      | 2.53MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 3   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 4   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 5   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| Total |       |     | -1    | 5      | 12.58MB | 0B           |                   |         |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+
+show column stats year_month_day;
++-----------+---------+------------------+--------+----------+----------+
+| Column    | Type    | #Distinct Values | #Nulls | Max Size | Avg Size |
++-----------+---------+------------------+--------+----------+----------+
+| id        | INT     | -1               | -1     | 4        | 4        |
+| val       | INT     | -1               | -1     | 4        | 4        |
+| zfill     | STRING  | -1               | -1     | -1       | -1       |
+| name      | STRING  | -1               | -1     | -1       | -1       |
+| assertion | BOOLEAN | -1               | -1     | 1        | 1        |
+| year      | INT     | 1                | 0      | 4        | 4        |
+| month     | INT     | 1                | 0      | 4        | 4        |
+| day       | INT     | 5                | 0      | 4        | 4        |
++-----------+---------+------------------+--------+----------+----------+
+
+compute stats year_month_day;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 5 partition(s) and 5 column(s). |
++-----------------------------------------+
+
+show table stats year_month_day;
++-------+-------+-----+--------+--------+---------+--------------+-------------------+---------+...
+| year  | month | day | #Rows  | #Files | Size    | Bytes Cached | Cache Replication | Format  |...
++-------+-------+-----+--------+--------+---------+--------------+-------------------+---------+...
+| 2013  | 12    | 1   | 93606  | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 2   | 94158  | 1      | 2.53MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 3   | 94122  | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 4   | 93559  | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 5   | 93845  | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| Total |       |     | 469290 | 5      | 12.58MB | 0B           |                   |         |...
++-------+-------+-----+--------+--------+---------+--------------+-------------------+---------+...
+
+show column stats year_month_day;
++-----------+---------+------------------+--------+----------+-------------------+
+| Column    | Type    | #Distinct Values | #Nulls | Max Size | Avg Size          |
++-----------+---------+------------------+--------+----------+-------------------+
+| id        | INT     | 511129           | -1     | 4        | 4                 |
+| val       | INT     | 364853           | -1     | 4        | 4                 |
+| zfill     | STRING  | 311430           | -1     | 6        | 6                 |
+| name      | STRING  | 471975           | -1     | 22       | 13.00160026550293 |
+| assertion | BOOLEAN | 2                | -1     | 1        | 1                 |
+| year      | INT     | 1                | 0      | 4        | 4                 |
+| month     | INT     | 1                | 0      | 4        | 4                 |
+| day       | INT     | 5                | 0      | 4        | 4                 |
++-----------+---------+------------------+--------+----------+-------------------+
+</codeblock>
+
+      <note>
+        Partitioned tables can grow so large that scanning the entire table, as the <codeph>COMPUTE STATS</codeph>
+        statement does, is impractical just to update the statistics for a new partition. The standard
+        <codeph>COMPUTE STATS</codeph> statement might take hours, or even days. That situation is where you switch
+        to using incremental statistics, a feature available in <keyword keyref="impala21_full"/> and higher.
+        See <xref href="impala_perf_stats.xml#perf_stats_incremental"/> for details about this feature
+        and the <codeph>COMPUTE INCREMENTAL STATS</codeph> syntax.
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="perf_stats_incremental">
+
+    <title id="incremental_stats">Overview of Incremental Statistics</title>
+
+    <conbody>
+
+      <p>
+        In Impala 2.1.0 and higher, you can use the syntax <codeph>COMPUTE INCREMENTAL STATS</codeph> and
+        <codeph>DROP INCREMENTAL STATS</codeph>. The <codeph>INCREMENTAL</codeph> clauses work with incremental
+        statistics, a specialized feature for partitioned tables that are large or frequently updated with new
+        partitions.
+      </p>
+
+      <p>
+        When you compute incremental statistics for a partitioned table, by default Impala only processes those
+        partitions that do not yet have incremental statistics. By processing only newly added partitions, you can
+        keep statistics up to date for large partitioned tables, without incurring the overhead of reprocessing the
+        entire table each time.
+      </p>
+
+      <p>
+        You can also compute or drop statistics for a single partition by including a <codeph>PARTITION</codeph>
+        clause in the <codeph>COMPUTE INCREMENTAL STATS</codeph> or <codeph>DROP INCREMENTAL STATS</codeph>
+        statement.
+      </p>
+
+      <p>
+        The metadata for incremental statistics is handled differently from the original style of statistics:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            If you have an existing partitioned table for which you have already computed statistics, issuing
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> without a partition clause causes Impala to rescan the
+            entire table. Once the incremental statistics are computed, any future <codeph>COMPUTE INCREMENTAL
+            STATS</codeph> statements only scan any new partitions and any partitions where you performed
+            <codeph>DROP INCREMENTAL STATS</codeph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>SHOW TABLE STATS</codeph> and <codeph>SHOW PARTITIONS</codeph> statements now include an
+            additional column showing whether incremental statistics are available for each column. A partition
+            could already be covered by the original type of statistics based on a prior <codeph>COMPUTE
+            STATS</codeph> statement, as indicated by a value other than <codeph>-1</codeph> under the
+            <codeph>#Rows</codeph> column. Impala query planning uses either kind of statistics when available.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> takes more time than <codeph>COMPUTE STATS</codeph> for the
+            same volume of data. Therefore it is most suitable for tables with large data volume where new
+            partitions are added frequently, making it impractical to run a full <codeph>COMPUTE STATS</codeph>
+            operation for each new partition. For unpartitioned tables, or partitioned tables that are loaded once
+            and not updated with new partitions, use the original <codeph>COMPUTE STATS</codeph> syntax.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> uses some memory in the <cmdname>catalogd</cmdname> process,
+            proportional to the number of partitions and number of columns in the applicable table. The memory
+            overhead is approximately 400 bytes for each column in each partition. This memory is reserved in the
+            <cmdname>catalogd</cmdname> daemon, the <cmdname>statestored</cmdname> daemon, and in each instance of
+            the <cmdname>impalad</cmdname> daemon.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            In cases where new files are added to an existing partition, issue a <codeph>REFRESH</codeph> statement
+            for the table, followed by a <codeph>DROP INCREMENTAL STATS</codeph> and <codeph>COMPUTE INCREMENTAL
+            STATS</codeph> sequence for the changed partition.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>DROP INCREMENTAL STATS</codeph> statement operates only on a single partition at a time. To
+            remove statistics (whether incremental or not) from all partitions of a table, issue a <codeph>DROP
+            STATS</codeph> statement with no <codeph>INCREMENTAL</codeph> or <codeph>PARTITION</codeph> clauses.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        The following considerations apply to incremental statistics when the structure of an existing table is
+        changed (known as <term>schema evolution</term>):
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to drop a column, the existing statistics remain
+            valid and <codeph>COMPUTE INCREMENTAL STATS</codeph> does not rescan any partitions.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to add a column, Impala rescans all partitions and
+            fills in the appropriate column-level values the next time you run <codeph>COMPUTE INCREMENTAL
+            STATS</codeph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to change the data type of a column, Impala
+            rescans all partitions and fills in the appropriate column-level values the next time you run
+            <codeph>COMPUTE INCREMENTAL STATS</codeph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to change the file format of a table, the existing
+            statistics remain valid and a subsequent <codeph>COMPUTE INCREMENTAL STATS</codeph> does not rescan any
+            partitions.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        See <xref href="impala_compute_stats.xml#compute_stats"/> and
+        <xref href="impala_drop_stats.xml#drop_stats"/> for syntax details.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="perf_stats_computing">
+    <title>Generating Table and Column Statistics (COMPUTE STATS Statement)</title>
+    <conbody>
+
+      <p>
+        To gather table statistics after loading data into a table or partition, you typically use the
+        <codeph>COMPUTE STATS</codeph> statement. This statement is available in Impala 1.2.2 and higher.
+        It gathers both table statistics and column statistics for all columns in a single operation.
+        For large partitioned tables, where you frequently need to update statistics and it is impractical
+        to scan the entire table each time, use the syntax <codeph>COMPUTE INCREMENTAL STATS</codeph>,
+        which is available in <keyword keyref="impala21_full"/> and higher.
+      </p>
+
+      <p>
+        If you use Hive as part of your ETL workflow, you can also use Hive to generate table and
+        column statistics. You might need to do extra configuration within Hive itself, the metastore,
+        or even set up a separate database to hold Hive-generated statistics. You might need to run
+        multiple statements to generate all the necessary statistics. Therefore, prefer the
+        Impala <codeph>COMPUTE STATS</codeph> statement where that technique is practical.
+        For details about collecting statistics through Hive, see
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/StatsDev" scope="external" format="html">the Hive wiki</xref>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+
+<!-- Commenting out over-detailed Hive instructions as part of stats reorg.
+        <li>
+          Issue an <codeph>ANALYZE TABLE</codeph> statement in Hive, for the entire table or a specific partition.
+<codeblock>ANALYZE TABLE <varname>tablename</varname> [PARTITION(<varname>partcol1</varname>[=<varname>val1</varname>], <varname>partcol2</varname>[=<varname>val2</varname>], ...)] COMPUTE STATISTICS [NOSCAN];</codeblock>
+          For example, to gather statistics for a non-partitioned table:
+<codeblock>ANALYZE TABLE customer COMPUTE STATISTICS;</codeblock>
+          To gather statistics for a <codeph>store</codeph> table partitioned by state and city, and both of its
+          partitions:
+<codeblock>ANALYZE TABLE store PARTITION(s_state, s_county) COMPUTE STATISTICS;</codeblock>
+          To gather statistics for the <codeph>store</codeph> table and only the partitions for California:
+<codeblock>ANALYZE TABLE store PARTITION(s_state='CA', s_county) COMPUTE STATISTICS;</codeblock>
+        </li>
+
+        <li>
+          Load the data through the <codeph>INSERT OVERWRITE</codeph> statement in Hive, while the Hive setting
+          <b>hive.stats.autogather</b> is enabled.
+        </li>
+
+      </ul>
+-->
+
+      <p rev="2.0.1">
+<!-- Additional info as a result of IMPALA-1420 -->
+<!-- Keep checking if https://issues.apache.org/jira/browse/HIVE-8648 ever gets fixed and when that fix makes it into a CDH release. -->
+        For your very largest tables, you might find that <codeph>COMPUTE STATS</codeph> or even <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        take so long to scan the data that it is impractical to use them regularly. In such a case, after adding a partition or inserting new data,
+        you can update just the number of rows property through an <codeph>ALTER TABLE</codeph> statement.
+        See <xref href="impala_perf_stats.xml#perf_table_stats_manual"/> for details.
+        Because the column statistics might be left in a stale state, do not use this technique as a replacement
+        for <codeph>COMPUTE STATS</codeph>. Only use this technique if all other means of collecting statistics are impractical, or as a
+        low-overhead operation that you run in between periodic <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph> operations.
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="perf_stats_checking">
+
+    <title>Detecting Missing Statistics</title>
+
+    <conbody>
+
+      <p>
+        You can check whether a specific table has statistics using the <codeph>SHOW TABLE STATS</codeph> statement
+        (for any table) or the <codeph>SHOW PARTITIONS</codeph> statement (for a partitioned table). Both
+        statements display the same information. If a table or a partition does not have any statistics, the
+        <codeph>#Rows</codeph> field contains <codeph>-1</codeph>. Once you compute statistics for the table or
+        partition, the <codeph>#Rows</codeph> field changes to an accurate value.
+      </p>
+
+      <p>
+        The following example shows a table that initially does not have any statistics. The <codeph>SHOW TABLE
+        STATS</codeph> statement displays different values for <codeph>#Rows</codeph> before and after the
+        <codeph>COMPUTE STATS</codeph> operation.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table no_stats (x int);
+[localhost:21000] &gt; show table stats no_stats;
++-------+--------+------+--------------+--------+-------------------+
+| #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+--------+------+--------------+--------+-------------------+
+| -1    | 0      | 0B   | NOT CACHED   | TEXT   | false             |
++-------+--------+------+--------------+--------+-------------------+
+[localhost:21000] &gt; compute stats no_stats;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 1 column(s). |
++-----------------------------------------+
+[localhost:21000] &gt; show table stats no_stats;
++-------+--------+------+--------------+--------+-------------------+
+| #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+--------+------+--------------+--------+-------------------+
+| 0     | 0      | 0B   | NOT CACHED   | TEXT   | false             |
++-------+--------+------+--------------+--------+-------------------+
+</codeblock>
+
+      <p>
+        The following example shows a similar progression with a partitioned table. Initially,
+        <codeph>#Rows</codeph> is <codeph>-1</codeph>. After a <codeph>COMPUTE STATS</codeph> operation,
+        <codeph>#Rows</codeph> changes to an accurate value. Any newly added partition starts with no statistics,
+        meaning that you must collect statistics after adding a new partition.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table no_stats_partitioned (x int) partitioned by (year smallint);
+[localhost:21000] &gt; show table stats no_stats_partitioned;
++-------+-------+--------+------+--------------+--------+-------------------+
+| year  | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+-------+--------+------+--------------+--------+-------------------+
+| Total | -1    | 0      | 0B   | 0B           |        |                   |
++-------+-------+--------+------+--------------+--------+-------------------+
+[localhost:21000] &gt; show partitions no_stats_partitioned;
++-------+-------+--------+------+--------------+--------+-------------------+
+| year  | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+-------+--------+------+--------------+--------+-------------------+
+| Total | -1    | 0      | 0B   | 0B           |        |                   |
++-------+-------+--------+------+--------------+--------+-------------------+
+[localhost:21000] &gt; alter table no_stats_partitioned add partition (year=2013);
+[localhost:21000] &gt; compute stats no_stats_partitioned;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 1 column(s). |
++-----------------------------------------+
+[localhost:21000] &gt; alter table no_stats_partitioned add partition (year=2014);
+[localhost:21000] &gt; show partitions no_stats_partitioned;
++-------+-------+--------+------+--------------+--------+-------------------+
+| year  | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+-------+--------+------+--------------+--------+-------------------+
+| 2013  | 0     | 0      | 0B   | NOT CACHED   | TEXT   | false             |
+| 2014  | -1    | 0      | 0B   | NOT CACHED   | TEXT   | false             |
+| Total | 0     | 0      | 0B   | 0B           |        |                   |
++-------+-------+--------+------+--------------+--------+-------------------+
+</codeblock>
+
+      <note>
+        Because the default <codeph>COMPUTE STATS</codeph> statement creates and updates statistics for all
+        partitions in a table, if you expect to frequently add new partitions, use the <codeph>COMPUTE INCREMENTAL
+        STATS</codeph> syntax instead, which lets you compute stats for a single specified partition, or only for
+        those partitions that do not already have incremental stats.
+      </note>
+
+      <p>
+        If checking each individual table is impractical, due to a large number of tables or views that hide the
+        underlying base tables, you can also check for missing statistics for a particular query. Use the
+        <codeph>EXPLAIN</codeph> statement to preview query efficiency before actually running the query. Use the
+        query profile output available through the <codeph>PROFILE</codeph> command in
+        <cmdname>impala-shell</cmdname> or the web UI to verify query execution and timing after running the query.
+        Both the <codeph>EXPLAIN</codeph> plan and the <codeph>PROFILE</codeph> output display a warning if any
+        tables or partitions involved in the query do not have statistics.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table no_stats (x int);
+[localhost:21000] &gt; explain select count(*) from no_stats;
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=10.00MB VCores=1                           |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| incremental_stats.no_stats                                                         |
+|                                                                                    |
+| 03:AGGREGATE [FINALIZE]                                                            |
+| |  output: count:merge(*)                                                          |
+| |                                                                                  |
+| 02:EXCHANGE [UNPARTITIONED]                                                        |
+| |                                                                                  |
+| 01:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+| 00:SCAN HDFS [incremental_stats.no_stats]                                          |
+|    partitions=1/1 files=0 size=0B                                                  |
++------------------------------------------------------------------------------------+
+</codeblock>
+
+      <p>
+        Because Impala uses the <term>partition pruning</term> technique when possible to only evaluate certain
+        partitions, if you have a partitioned table with statistics for some partitions and not others, whether or
+        not the <codeph>EXPLAIN</codeph> statement shows the warning depends on the actual partitions used by the
+        query. For example, you might see warnings or not for different queries against the same table:
+      </p>
+
+<codeblock>-- No warning because all the partitions for the year 2012 have stats.
+EXPLAIN SELECT ... FROM t1 WHERE year = 2012;
+
+-- Missing stats warning because one or more partitions in this range
+-- do not have stats.
+EXPLAIN SELECT ... FROM t1 WHERE year BETWEEN 2006 AND 2009;
+</codeblock>
+
+      <p>
+        To confirm if any partitions at all in the table are missing statistics, you might explain a query that
+        scans the entire table, such as <codeph>SELECT COUNT(*) FROM <varname>table_name</varname></codeph>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="perf_stats_collecting">
+
+    <title>Keeping Statistics Up to Date</title>
+
+    <conbody>
+
+      <p>
+        When the contents of a table or partition change significantly, recompute the stats for the relevant table
+        or partition. The degree of change that qualifies as <q>significant</q> varies, depending on the absolute
+        and relative sizes of the tables. Typically, if you add more than 30% more data to a table, it is
+        worthwhile to recompute stats, because the differences in number of rows and number of distinct values
+        might cause Impala to choose a different join order when that table is used in join queries. This guideline
+        is most important for the largest tables. For example, adding 30% new data to a table containing 1 TB has a
+        greater effect on join order than adding 30% to a table containing only a few megabytes, and the larger
+        table has a greater effect on query performance if Impala chooses a suboptimal join order as a result of
+        outdated statistics.
+      </p>
+
+      <p>
+        If you reload a complete new set of data for a table, but the number of rows and number of distinct values
+        for each column is relatively unchanged from before, you do not need to recompute stats for the table.
+      </p>
+
+      <p>
+        If the statistics for a table are out of date, and the table's large size makes it impractical to recompute
+        new stats immediately, you can use the <codeph>DROP STATS</codeph> statement to remove the obsolete
+        statistics, making it easier to identify tables that need a new <codeph>COMPUTE STATS</codeph> operation.
+      </p>
+
+      <p>
+        For a large partitioned table, consider using the incremental stats feature available in Impala 2.1.0 and
+        higher, as explained in <xref href="impala_perf_stats.xml#perf_stats_incremental"/>. If you add a new
+        partition to a table, it is worthwhile to recompute incremental stats, because the operation only scans the
+        data for that one new partition.
+      </p>
+    </conbody>
+  </concept>
+
+<!-- Might deserve its own conceptual topic at some point. -->
+
+  <concept audience="Cloudera" rev="1.2.2" id="perf_stats_joins">
+
+    <title>How Statistics Are Used in Join Queries</title>
+
+    <conbody>
+
+      <p></p>
+    </conbody>
+  </concept>
+
+<!-- Might deserve its own conceptual topic at some point. -->
+
+  <concept audience="Cloudera" rev="1.2.2" id="perf_stats_inserts">
+
+    <title>How Statistics Are Used in INSERT Operations</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/insert_hints"/>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.2" id="perf_table_stats_manual">
+
+    <title>Setting the NUMROWS Value Manually through ALTER TABLE</title>
+
+    <conbody>
+
+      <p>
+        The most crucial piece of data in all the statistics is the number of rows in the table (for an
+        unpartitioned or partitioned table) and for each partition (for a partitioned table). The <codeph>COMPUTE STATS</codeph>
+        statement always gathers statistics about all columns, as well as overall table statistics. If it is not
+        practical to do a full <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        operation after adding a partition or inserting data, or if you can see that Impala would produce a more
+        efficient plan if the number of rows was different, you can manually set the number of rows through an
+        <codeph>ALTER TABLE</codeph> statement:
+      </p>
+
+<codeblock>
+-- Set total number of rows. Applies to both unpartitioned and partitioned tables.
+alter table <varname>table_name</varname> set tblproperties('numRows'='<varname>new_value</varname>', 'STATS_GENERATED_VIA_STATS_TASK'='true');
+
+-- Set total number of rows for a specific partition. Applies to partitioned tables only.
+-- You must specify all the partition key columns in the PARTITION clause.
+alter table <varname>table_name</varname> partition (<varname>keycol1</varname>=<varname>val1</varname>,<varname>keycol2</varname>=<varname>val2</varname>...) set tblproperties('numRows'='<varname>new_value</varname>', 'STATS_GENERATED_VIA_STATS_TASK'='true');
+</codeblock>
+
+      <p>
+        This statement avoids re-scanning any data files. (The requirement to include the <codeph>STATS_GENERATED_VIA_STATS_TASK</codeph> property is relatively new, as a
+        result of the issue <xref href="https://issues.apache.org/jira/browse/HIVE-8648" scope="external" format="html">HIVE-8648</xref>
+        for the Hive metastore.)
+      </p>
+
+<codeblock conref="../shared/impala_common.xml#common/set_numrows_example"/>
+
+      <p>
+        For a partitioned table, update both the per-partition number of rows and the number of rows for the whole
+        table:
+      </p>
+
+<codeblock conref="../shared/impala_common.xml#common/set_numrows_partitioned_example"/>
+
+      <p>
+        In practice, the <codeph>COMPUTE STATS</codeph> statement, or <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        for a partitioned table, should be fast and convenient enough that this technique is only useful for the very
+        largest partitioned tables.
+        <!--
+        It is most useful as a workaround for in case of performance issues where you might adjust the <codeph>numRows</codeph> value higher
+        or lower to produce the ideal join order.
+        -->
+        <!-- Following wording is duplicated from earlier. Consider conref'ing. -->
+        Because the column statistics might be left in a stale state, do not use this technique as a replacement
+        for <codeph>COMPUTE STATS</codeph>. Only use this technique if all other means of collecting statistics are impractical, or as a
+        low-overhead operation that you run in between periodic <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph> operations.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="perf_column_stats_manual" rev="2.6.0 IMPALA-3369">
+    <title>Setting Column Stats Manually through ALTER TABLE</title>
+    <conbody>
+      <p>
+        In <keyword keyref="impala26_full"/> and higher, you can also use the <codeph>SET COLUMN STATS</codeph>
+        clause of <codeph>ALTER TABLE</codeph> to manually set or change column statistics.
+        Only use this technique in cases where it is impractical to run
+        <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        frequently enough to keep up with data changes for a huge table.
+      </p>
+      <p conref="../shared/impala_common.xml#common/set_column_stats_example"/>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.2" id="perf_stats_examples">
+
+    <title>Examples of Using Table and Column Statistics with Impala</title>
+
+    <conbody>
+
+      <p>
+        The following examples walk through a sequence of <codeph>SHOW TABLE STATS</codeph>, <codeph>SHOW COLUMN
+        STATS</codeph>, <codeph>ALTER TABLE</codeph>, and <codeph>SELECT</codeph> and <codeph>INSERT</codeph>
+        statements to illustrate various aspects of how Impala uses statistics to help optimize queries.
+      </p>
+
+      <p>
+        This example shows table and column statistics for the <codeph>STORE</codeph> column used in the
+        <xref href="http://www.tpc.org/tpcds/" scope="external" format="html">TPC-DS benchmarks for decision
+        support</xref> systems. It is a tiny table holding data for 12 stores. Initially, before any statistics are
+        gathered by a <codeph>COMPUTE STATS</codeph> statement, most of the numeric fields show placeholder values
+        of -1, indicating that the figures are unknown. The figures that are filled in are values that are easily
+        countable or deducible at the physical level, such as the number of files, total data size of the files,
+        and the maximum and average sizes for data types that have a constant size such as <codeph>INT</codeph>,
+        <codeph>FLOAT</codeph>, and <codeph>TIMESTAMP</codeph>.
+      </p>
+
+<codeblock>[localhost:21000] &gt; show table stats store;
++-------+--------+--------+--------+
+| #Rows | #Files | Size   | Format |
++-------+--------+--------+--------+
+| -1    | 1      | 3.08KB | TEXT   |
++-------+--------+--------+--------+
+Returned 1 row(s) in 0.03s
+[localhost:21000] &gt; show column stats store;
++--------------------+-----------+------------------+--------+----------+----------+
+| Column             | Type      | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------------------+-----------+------------------+--------+----------+----------+
+| s_store_sk         | INT       | -1               | -1     | 4        | 4        |
+| s_store_id         | STRING    | -1               | -1     | -1       | -1       |
+| s_rec_start_date   | TIMESTAMP | -1               | -1     | 16       | 16       |
+| s_rec_end_date     | TIMESTAMP | -1               | -1     | 16       | 16       |
+| s_closed_date_sk   | INT       | -1               | -1     | 4        | 4        |
+| s_store_name       | STRING    | -1               | -1     | -1       | -1       |
+| s_number_employees | INT       | -1               | -1     | 4        | 4        |
+| s_floor_space      | INT       | -1               | -1     | 4        | 4        |
+| s_hours            | STRING    | -1               | -1     | -1       | -1       |
+| s_manager          | STRING    | -1               | -1     | -1       | -1       |
+| s_market_id        | INT       | -1               | -1     | 4        | 4        |
+| s_geography_class  | STRING    | -1               | -1     | -1       | -1       |
+| s_market_desc      | STRING    | -1               | -1     | -1       | -1       |
+| s_market_manager   | STRING    | -1               | -1     | -1       | -1       |
+| s_division_id      | INT       | -1               | -1     | 4        | 4        |
+| s_division_name    | STRING    | -1               | -1     | -1       | -1       |
+| s_company_id       | INT       | -1               | -1     | 4        | 4        |
+| s_company_name     | STRING    | -1               | -1     | -1       | -1       |
+| s_street_number    | STRING    | -1               | -1     | -1       | -1       |
+| s_street_name      | STRING    | -1               | -1     | -1       | -1       |
+| s_street_type      | STRING    | -1               | -1     | -1       | -1       |
+| s_suite_number     | STRING    | -1               | -1     | -1       | -1       |
+| s_city             | STRING    | -1               | -1     | -1       | -1       |
+| s_county           | STRING    | -1               | -1     | -1       | -1       |
+| s_state            | STRING    | -1               | -1     | -1       | -1       |
+| s_zip              | STRING    | -1               | -1     | -1       | -1       |
+| s_country          | STRING    | -1               | -1     | -1       | -1       |
+| s_gmt_offset       | FLOAT     | -1               | -1     | 4        | 4        |
+| s_tax_percentage   | FLOAT     | -1               | -1     | 4        | 4        |
++--------------------+-----------+------------------+--------+----------+----------+
+Returned 29 row(s) in 0.04s</codeblock>
+
+      <p>
+        With the Hive <codeph>ANALYZE TABLE</codeph> statement for column statistics, you had to specify each
+        column for which to gather statistics. The Impala <codeph>COMPUTE STATS</codeph> statement automatically
+        gathers statistics for all columns, because it reads through the entire table relatively quickly and can
+        efficiently compute the values for all the columns. This example shows how after running the
+        <codeph>COMPUTE STATS</codeph> statement, statistics are filled in for both the table and all its columns:
+      </p>
+
+<codeblock>[localhost:21000] &gt; compute stats store;
++------------------------------------------+
+| summary                                  |
++------------------------------------------+
+| Updated 1 partition(s) and 29 column(s). |
++------------------------------------------+
+Returned 1 row(s) in 1.88s
+[localhost:21000] &gt; show table stats store;
++-------+--------+--------+--------+
+| #Rows | #Files | Size   | Format |
++-------+--------+--------+--------+
+| 12    | 1      | 3.08KB | TEXT   |
++-------+--------+--------+--------+
+Returned 1 row(s) in 0.02s
+[localhost:21000] &gt; show column stats store;
++--------------------+-----------+------------------+--------+----------+-------------------+
+| Column             | Type      | #Distinct Values | #Nulls | Max Size | Avg Size          |
++--------------------+-----------+------------------+--------+----------+-------------------+
+| s_store_sk         | INT       | 12               | -1     | 4        | 4                 |
+| s_store_id         | STRING    | 6                | -1     | 16       | 16                |
+| s_rec_start_date   | TIMESTAMP | 4                | -1     | 16       | 16                |
+| s_rec_end_date     | TIMESTAMP | 3                | -1     | 16       | 16                |
+| s_closed_date_sk   | INT       | 3                | -1     | 4        | 4                 |
+| s_store_name       | STRING    | 8                | -1     | 5        | 4.25              |
+| s_number_employees | INT       | 9                | -1     | 4        | 4                 |
+| s_floor_space      | INT       | 10               | -1     | 4        | 4                 |
+| s_hours            | STRING    | 2                | -1     | 8        | 7.083300113677979 |
+| s_manager          | STRING    | 7                | -1     | 15       | 12                |
+| s_market_id        | INT       | 7                | -1     | 4        | 4                 |
+| s_geography_class  | STRING    | 1                | -1     | 7        | 7                 |
+| s_market_desc      | STRING    | 10               | -1     | 94       | 55.5              |
+| s_market_manager   | STRING    | 7                | -1     | 16       | 14                |
+| s_division_id      | INT       | 1                | -1     | 4        | 4                 |
+| s_division_name    | STRING    | 1                | -1     | 7        | 7                 |
+| s_company_id       | INT       | 1                | -1     | 4        | 4                 |
+| s_company_name     | STRING    | 1                | -1     | 7        | 7                 |
+| s_street_number    | STRING    | 9                | -1     | 3        | 2.833300113677979 |
+| s_street_name      | STRING    | 12               | -1     | 11       | 6.583300113677979 |
+| s_street_type      | STRING    | 8                | -1     | 9        | 4.833300113677979 |
+| s_suite_number     | STRING    | 11               | -1     | 9        | 8.25              |
+| s_city             | STRING    | 2                | -1     | 8        | 6.5               |
+| s_county           | STRING    | 1                | -1     | 17       | 17                |
+| s_state            | STRING    | 1                | -1     | 2        | 2                 |
+| s_zip              | STRING    | 2                | -1     | 5        | 5                 |
+| s_country          | STRING    | 1                | -1     | 13       | 13                |
+| s_gmt_offset       | FLOAT     | 1                | -1     | 4        | 4                 |
+| s_tax_percentage   | FLOAT     | 5                | -1     | 4        | 4                 |
++--------------------+-----------+------------------+--------+----------+-------------------+
+Returned 29 row(s) in 0.04s</codeblock>
+
+      <p>
+        The following example shows how statistics are represented for a partitioned table. In this case, we have
+        set up a table to hold the world's most trivial census data, a single <codeph>STRING</codeph> field,
+        partitioned by a <codeph>YEAR</codeph> column. The table statistics include a separate entry for each
+        partition, plus final totals for the numeric fields. The column statistics include some easily deducible
+        facts for the partitioning column, such as the number of distinct values (the number of partition
+        subdirectories).
+<!-- and the number of <codeph>NULL</codeph> values (none in this case). -->
+      </p>
+
+<codeblock>localhost:21000] &gt; describe census;
++------+----------+---------+
+| name | type     | comment |
++------+----------+---------+
+| name | string   |         |
+| year | smallint |         |
++------+----------+---------+
+Returned 2 row(s) in 0.02s
+[localhost:21000] &gt; show table stats census;
++-------+-------+--------+------+---------+
+| year  | #Rows | #Files | Size | Format  |
++-------+-------+--------+------+---------+
+| 2000  | -1    | 0      | 0B   | TEXT    |
+| 2004  | -1    | 0      | 0B   | TEXT    |
+| 2008  | -1    | 0      | 0B   | TEXT    |
+| 2010  | -1    | 0      | 0B   | TEXT    |
+| 2011  | 0     | 1      | 22B  | TEXT    |
+| 2012  | -1    | 1      | 22B  | TEXT    |
+| 2013  | -1    | 1      | 231B | PARQUET |
+| Total | 0     | 3      | 275B |         |
++-------+-------+--------+------+---------+
+Returned 8 row(s) in 0.02s
+[localhost:21000] &gt; show column stats census;
++--------+----------+------------------+--------+----------+----------+
+| Column | Type     | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+----------+------------------+--------+----------+----------+
+| name   | STRING   | -1               | -1     | -1       | -1       |
+| year   | SMALLINT | 7                | -1     | 2        | 2        |
++--------+----------+------------------+--------+----------+----------+
+Returned 2 row(s) in 0.02s</codeblock>
+
+      <p>
+        The following example shows how the statistics are filled in by a <codeph>COMPUTE STATS</codeph> statement
+        in Impala.
+      </p>
+
+<codeblock>[localhost:21000] &gt; compute stats census;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 3 partition(s) and 1 column(s). |
++-----------------------------------------+
+Returned 1 row(s) in 2.16s
+[localhost:21000] &gt; show table stats census;
++-------+-------+--------+------+---------+
+| year  | #Rows | #Files | Size | Format  |
++-------+-------+--------+------+---------+
+| 2000  | -1    | 0      | 0B   | TEXT    |
+| 2004  | -1    | 0      | 0B   | TEXT    |
+| 2008  | -1    | 0      | 0B   | TEXT    |
+| 2010  | -1    | 0      | 0B   | TEXT    |
+| 2011  | 4     | 1      | 22B  | TEXT    |
+| 2012  | 4     | 1      | 22B  | TEXT    |
+| 2013  | 1     | 1      | 231B | PARQUET |
+| Total | 9     | 3      | 275B |         |
++-------+-------+--------+------+---------+
+Returned 8 row(s) in 0.02s
+[localhost:21000] &gt; show column stats census;
++--------+----------+------------------+--------+----------+----------+
+| Column | Type     | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+----------+------------------+--------+----------+----------+
+| name   | STRING   | 4                | -1     | 5        | 4.5      |
+| year   | SMALLINT | 7                | -1     | 2        | 2        |
++--------+----------+------------------+--------+----------+----------+
+Returned 2 row(s) in 0.02s</codeblock>
+
+      <p rev="1.4.0">
+        For examples showing how some queries work differently when statistics are available, see
+        <xref href="impala_perf_joins.xml#perf_joins_examples"/>. You can see how Impala executes a query
+        differently in each case by observing the <codeph>EXPLAIN</codeph> output before and after collecting
+        statistics. Measure the before and after query times, and examine the throughput numbers in before and
+        after <codeph>SUMMARY</codeph> or <codeph>PROFILE</codeph> output, to verify how much the improved plan
+        speeds up performance.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_testing.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_testing.xml b/docs/topics/impala_perf_testing.xml
new file mode 100644
index 0000000..d621556
--- /dev/null
+++ b/docs/topics/impala_perf_testing.xml
@@ -0,0 +1,175 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="performance_testing">
+
+  <title>Testing Impala Performance</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Logs"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Should reorg this topic to use nested topics, not sections. Some keywords like 'logs' buried in section titles. -->
+      <data name="Category" value="Sectionated Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Test to ensure that Impala is configured for optimal performance. If you have installed Impala without
+      Cloudera Manager, complete the processes described in this topic to help ensure a proper configuration. Even
+      if you installed Impala with Cloudera Manager, which automatically applies appropriate configurations, these
+      procedures can be used to verify that Impala is set up correctly.
+    </p>
+
+    <section id="checking_config_performance">
+
+      <title>Checking Impala Configuration Values</title>
+
+      <p>
+        You can inspect Impala configuration values by connecting to your Impala server using a browser.
+      </p>
+
+      <p>
+        <b>To check Impala configuration values:</b>
+      </p>
+
+      <ol>
+        <li>
+          Use a browser to connect to one of the hosts running <codeph>impalad</codeph> in your environment.
+          Connect using an address of the form
+          <codeph>http://<varname>hostname</varname>:<varname>port</varname>/varz</codeph>.
+          <note>
+            In the preceding example, replace <codeph>hostname</codeph> and <codeph>port</codeph> with the name and
+            port of your Impala server. The default port is 25000.
+          </note>
+        </li>
+
+        <li>
+          Review the configured values.
+          <p>
+            For example, to check that your system is configured to use block locality tracking information, you
+            would check that the value for <codeph>dfs.datanode.hdfs-blocks-metadata.enabled</codeph> is
+            <codeph>true</codeph>.
+          </p>
+        </li>
+      </ol>
+
+      <p id="p_31">
+        <b>To check data locality:</b>
+      </p>
+
+      <ol>
+        <li>
+          Execute a query on a dataset that is available across multiple nodes. For example, for a table named
+          <codeph>MyTable</codeph> that has a reasonable chance of being spread across multiple DataNodes:
+<codeblock>[impalad-host:21000] &gt; SELECT COUNT (*) FROM MyTable</codeblock>
+        </li>
+
+        <li>
+          After the query completes, review the contents of the Impala logs. You should find a recent message
+          similar to the following:
+<codeblock>Total remote scan volume = 0</codeblock>
+        </li>
+      </ol>
+
+      <p>
+        The presence of remote scans may indicate <codeph>impalad</codeph> is not running on the correct nodes.
+        This can be because some DataNodes do not have <codeph>impalad</codeph> running or it can be because the
+        <codeph>impalad</codeph> instance that is starting the query is unable to contact one or more of the
+        <codeph>impalad</codeph> instances.
+      </p>
+
+      <p>
+        <b>To understand the causes of this issue:</b>
+      </p>
+
+      <ol>
+        <li>
+          Connect to the debugging web server. By default, this server runs on port 25000. This page lists all
+          <codeph>impalad</codeph> instances running in your cluster. If there are fewer instances than you expect,
+          this often indicates some DataNodes are not running <codeph>impalad</codeph>. Ensure
+          <codeph>impalad</codeph> is started on all DataNodes.
+        </li>
+
+        <li>
+          <!-- To do:
+            There are other references to this tip about the "Impala daemon's hostname" elsewhere. Could reconcile, conref, or link.
+          -->
+          If you are using multi-homed hosts, ensure that the Impala daemon's hostname resolves to the interface on
+          which <codeph>impalad</codeph> is running. The hostname Impala is using is displayed when
+          <codeph>impalad</codeph> starts. To explicitly set the hostname, use the <codeph>--hostname</codeph> flag.
+        </li>
+
+        <li>
+          Check that <codeph>statestored</codeph> is running as expected. Review the contents of the state store
+          log to ensure all instances of <codeph>impalad</codeph> are listed as having connected to the state
+          store.
+        </li>
+      </ol>
+    </section>
+
+    <section id="checking_config_logs">
+
+      <title>Reviewing Impala Logs</title>
+
+      <p>
+        You can review the contents of the Impala logs for signs that short-circuit reads or block location
+        tracking are not functioning. Before checking logs, execute a simple query against a small HDFS dataset.
+        Completing a query task generates log messages using current settings. Information on starting Impala and
+        executing queries can be found in <xref href="impala_processes.xml#processes"/> and
+        <xref href="impala_impala_shell.xml#impala_shell"/>. Information on logging can be found in
+        <xref href="impala_logging.xml#logging"/>. Log messages and their interpretations are as follows:
+      </p>
+
+      <table>
+        <tgroup cols="2">
+          <colspec colname="1" colwidth="30*"/>
+          <colspec colname="2" colwidth="10*"/>
+          <thead>
+            <row>
+              <entry>
+                Log Message
+              </entry>
+              <entry>
+                Interpretation
+              </entry>
+            </row>
+          </thead>
+          <tbody>
+            <row>
+              <entry>
+                <p>
+<pre>Unknown disk id. This will negatively affect performance. Check your hdfs settings to enable block location metadata
+</pre>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Tracking block locality is not enabled.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+<pre>Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</pre>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Native checksumming is not enabled.
+                </p>
+              </entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+    </section>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_performance.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_performance.xml b/docs/topics/impala_performance.xml
new file mode 100644
index 0000000..e58270e
--- /dev/null
+++ b/docs/topics/impala_performance.xml
@@ -0,0 +1,191 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="performance">
+
+  <title>Tuning Impala for Performance</title>
+  <titlealts audience="PDF"><navtitle>Performance Tuning</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <!-- Like Impala Administration, this page has a fair bit of info already, but it could benefit from wiki-style embedded of intro text from those other pages. -->
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The following sections explain the factors affecting the performance of Impala features, and procedures for
+      tuning, monitoring, and benchmarking Impala queries and other SQL operations.
+    </p>
+
+    <p>
+      This section also describes techniques for maximizing Impala scalability. Scalability is tied to performance:
+      it means that performance remains high as the system workload increases. For example, reducing the disk I/O
+      performed by a query can speed up an individual query, and at the same time improve scalability by making it
+      practical to run more queries simultaneously. Sometimes, an optimization technique improves scalability more
+      than performance. For example, reducing memory usage for a query might not change the query performance much,
+      but might improve scalability by allowing more Impala queries or other kinds of jobs to run at the same time
+      without running out of memory.
+    </p>
+
+    <note>
+      <p>
+        Before starting any performance tuning or benchmarking, make sure your system is configured with all the
+        recommended minimum hardware requirements from <xref href="impala_prereqs.xml#prereqs_hardware"/> and
+        software settings from <xref href="impala_config_performance.xml#config_performance"/>.
+      </p>
+    </note>
+
+    <ul>
+      <li>
+        <xref href="impala_partitioning.xml#partitioning"/>. This technique physically divides the data based on
+        the different values in frequently queried columns, allowing queries to skip reading a large percentage of
+        the data in a table.
+      </li>
+
+      <li>
+        <xref href="impala_perf_joins.xml#perf_joins"/>. Joins are the main class of queries that you can tune at
+        the SQL level, as opposed to changing physical factors such as the file format or the hardware
+        configuration. The related topics <xref href="impala_perf_stats.xml#perf_column_stats"/> and
+        <xref href="impala_perf_stats.xml#perf_table_stats"/> are also important primarily for join performance.
+      </li>
+
+      <li>
+        <xref href="impala_perf_stats.xml#perf_table_stats"/> and
+        <xref href="impala_perf_stats.xml#perf_column_stats"/>. Gathering table and column statistics, using the
+        <codeph>COMPUTE STATS</codeph> statement, helps Impala automatically optimize the performance for join
+        queries, without requiring changes to SQL query statements. (This process is greatly simplified in Impala
+        1.2.2 and higher, because the <codeph>COMPUTE STATS</codeph> statement gathers both kinds of statistics in
+        one operation, and does not require any setup and configuration as was previously necessary for the
+        <codeph>ANALYZE TABLE</codeph> statement in Hive.)
+      </li>
+
+      <li>
+        <xref href="impala_perf_testing.xml#performance_testing"/>. Do some post-setup testing to ensure Impala is
+        using optimal settings for performance, before conducting any benchmark tests.
+      </li>
+
+      <li>
+        <xref href="impala_perf_benchmarking.xml#perf_benchmarks"/>. The configuration and sample data that you use
+        for initial experiments with Impala is often not appropriate for doing performance tests.
+      </li>
+
+      <li>
+        <xref href="impala_perf_resources.xml#mem_limits"/>. The more memory Impala can utilize, the better query
+        performance you can expect. In a cluster running other kinds of workloads as well, you must make tradeoffs
+        to make sure all Hadoop components have enough memory to perform well, so you might cap the memory that
+        Impala can use.
+      </li>
+
+      <li rev="1.2" audience="Cloudera">
+        <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/>. Impala can use the HDFS caching feature to pin
+        frequently accessed data in memory, reducing disk I/O.
+      </li>
+
+      <li rev="2.2.0">
+        <xref href="impala_s3.xml#s3"/>. Queries against data stored in the Amazon Simple Storage Service (S3)
+        have different performance characteristics than when the data is stored in HDFS.
+      </li>
+    </ul>
+
+    <p outputclass="toc"/>
+
+    <p conref="../shared/impala_common.xml#common/cookbook_blurb"/>
+
+  </conbody>
+
+<!-- Empty/hidden stub sections that might be worth expanding later. -->
+
+  <concept id="perf_network" audience="Cloudera">
+
+    <title>Network Traffic</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_partition_schema" audience="Cloudera">
+
+    <title>Designing Partitioned Tables</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_partition_query" audience="Cloudera">
+
+    <title>Queries on Partitioned Tables</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_monitoring" audience="Cloudera">
+
+    <title>Monitoring Performance through the Impala Web Interface</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Monitoring"/>
+    </metadata>
+  </prolog>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_query_coord" audience="Cloudera">
+
+    <title>Query Coordination</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_bottlenecks" audience="Cloudera">
+
+    <title>Performance Bottlenecks</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_long_queries" audience="Cloudera">
+
+    <title>Managing Long-Running Queries</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_load" audience="Cloudera">
+
+    <title>Performance Considerations for Loading Data</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_file_formats" audience="Cloudera">
+
+    <title>Performance Considerations for File Formats</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_compression" audience="Cloudera">
+
+    <title>Performance Considerations for Compression</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_codegen" audience="Cloudera">
+
+    <title>Native Code Generation</title>
+
+    <conbody/>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_planning.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_planning.xml b/docs/topics/impala_planning.xml
new file mode 100644
index 0000000..f103ab8
--- /dev/null
+++ b/docs/topics/impala_planning.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="planning">
+
+  <title>Planning for Impala Deployment</title>
+  <titlealts audience="PDF"><navtitle>Deployment Planning</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Deploying"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">planning</indexterm>
+      Before you set up Impala in production, do some planning to make sure that your hardware setup has sufficient
+      capacity, that your cluster topology is optimal for Impala queries, and that your schema design and ETL
+      processes follow the best practices for Impala.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+</concept>


Mime
View raw message