trafodion-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dbirds...@apache.org
Subject [1/5] incubator-trafodion git commit: Replaced logo, added drafts of new manuals, updated SQL Reference, corrected issues in other manuals.
Date Thu, 03 Mar 2016 20:13:14 GMT
Repository: incubator-trafodion
Updated Branches:
  refs/heads/master 1824c7e3c -> a717a3e63


http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc
----------------------------------------------------------------------
diff --git a/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc b/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc
new file mode 100644
index 0000000..112b0ee
--- /dev/null
+++ b/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc
@@ -0,0 +1,159 @@
+////
+/**
+* @@@ START COPYRIGHT @@@
+*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements.  See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership.  The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License.  You may obtain a copy of the License at
+*
+*   http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied.  See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*
+* @@@ END COPYRIGHT @@@
+  */
+////
+
+[[troubleshoot]]
+= Troubleshoot
+== Improving Throughput 
+
+=== Tuplelists or Rowsets
+
+When Tuplelists or Rowsets are used as the data source, performance typically increases with
the
+number of rows in the Tuplelist or Rowset. Performance peaks at some value for the number
of rows
+and remain more or less steady after that. This peak value depends on row size.
+Typically a value in the range of 100 to few thousand is reasonable.
+
+=== Native HBase Tables
+
+When native HBase tables are used as the data source, it is important to override the default
value
+for the attribute HBASE_MAX_COLUMN_VALUE_LENGTH (columnwise mode) or HBASE_MAX_COLUMN_INFO_LENGTH
(rowwise mode)
+and set the value to the maximum for the table being used as the source.
+The default values may be too large.
+
+=== Hive Tables
+
+When Hive tables are used as the data source, it is important to override the default value
for the
+attribute HIVE_MAX_STRING_LENGTH when the Hive source table has columns of type string. Please
set the
+value to the length of the longest string in the Hive table.
+
+To determine that length, run this query from a Hive shell:
+
+```
+SELECT MAX(LENGTH(<col-name>)) FROM <hive-tab-name>;
+```
+
+If the query returns a value less than the current HIVE_MAX_STRING_LENGTH, then you need
to
+increase that value and retry. If the query  returns a value that is far less than the current
+HIVE_MAX_STRING_LENGTH, then you can achieve better performance by reducing the value. 
+An approximate value can be used, too. The Trafodion default of 32000 may be too generous
in some cases.
+
+== Checking Plan Quality
+
+It is good practice to check the quality of the plan generated by the SQL compiler before
executing a
+data loading statement that may take a long time to complete. 
+
+* For INSERT and UPSERT USING LOAD statements, use the EXPLAIN statement, which is described
in the 
+http://trafodion.apache.org/docs/sql_reference/index.html#explain_statement[Trafodion SQL
Reference Manual].
+* For the LOAD statement, which is implemented as a utility operator (that is, a collection
of secondary SQL statements),
+use the following SQL statements to see the plan that it uses to add data to the target table:
+
+```
+CONTROL QUERY DEFAULT COMP_BOOL_226 'ON' ;
+PREPARE s1 FROM LOAD TRANSFORM INTO <target-table> <select-query-used-as-source>
;
+EXPLAIN OPTIONS 'f' s1 ;
+```
+
+A typical problem with the plan is that the scan is not parallel enough. For Trafodion tables,
you can address this
+issue with the default attribute, PARALLEL_NUM_ESPS.  Using this attribute, a Trafodion scan
can be parallelized to
+as many number of SALT partitions that are defined for the table. For Hive source tables,
the default attributes, 
+HIVE_NUM_ESPS_PER_DATANODE and HIVE_MIN_BYTES_PER_ESP_PARTITION, can be used to adjust the
degree of parallelism.
+
+== UPDATE STATISTICS Times Out During Sampling
+
+Sampling in update statistics is implemented using the HBase Random RowFilter. For very large
tables with several
+billion rows, the sampling ratio required to get a sample of one million rows is very small.
This can result in
+HBase client connection timeout errors since there may be no row returned by a RegionServer
for an extended period of time.
+
+You can avoid this problem by:
+
+* Choosing a sampling percentage higher than the default setting of 1 million rows for large
tables.
++
+For example, suppose table T has one billion rows. Use the following UPDATE STATISTICS statement
to sample a million rows,
+or approximately one-tenth of one percent of the total rows:
++
+```
+UPDATE STATISTICS FOR TABLE t ON EVERY COLUMN SAMPLE;
+```
++
+To sample one percent of the rows, regardless of the table size, you must explicitly state
the sampling rate as follows:
++
+```
+UPDATE STATISTICS FOR TABLE t ON EVERY COLUMN SAMPLE RANDOM 1 PERCENT;
+```
+
+* Setting `hbase.rpc.timeout` to a higher value than currently specified in the HBase settings.
+
+== Index Creation Takes Too Long
+
+When creating an index, all rows of the Trafodion table must be scanned and a subset of columns
is returned to the client.
+This can take a while to complete.  If there is a Hive table with the same data as the Trafodion
table being scanned, then
+you can specify the default attribute, USE_HIVE_SOURCE. This causes the Hive table to be
used as the source creating the index. 
+
+NOTE: The name of the Hive table must use the Trafodion table name as its prefix.
+For example, if the Trafodion table is TRAFODION.SCH.DEMO, then the Hive 
+table name can be DEMO_SRC. In this case, set the attribute as follows: 
++
+```
+CONTROL QUERY DEFAULT USE_HIVE_SOURCE '_SRC' ;
+CREATE INDEX demo_ix ON sch.demo(name) ;
+```
+
+[[large-deletes]]
+== Large Deletes Take Too Long or Error Out
+
+If a large number of rows is either updated or deleted in a single SQL statement, then it
is likely that the
+statement does not complete successfully.
+
+Deleting or updating more than 10,000 rows with a single statement is not recommended. Instead,
a large delete
+or update should be broken up into multiple statements  each affecting less than 10,000*n
rows, if possible.
+`n` is number of nodes in the cluster.
+
+== Large UPSERT USING LOAD On a Table With Index Errors Out
+
+UPSERT USING LOAD automatically reverts to a transactional UPSERT when used on a table with
an index. This causes Trafodion
+to run into the limitation discusses in <<large-deletes,Large Deletes Take Too Long
or Error Out>> above:
+no more than 10,000*n rows (n = number of nodes) can be affected in a single statement.
+
+*Workaround*: The UPSERT USING LOAD operation can be placed in a LOAD statement as shown
below. The LOAD statement disables
+indexes on the table before the UPSERT USING LOAD starts. Once the UPSERT USING LOAD  completes
indexes are populated by
+the LOAD statement. 
+
+```
+LOAD WITH UPSERT USING LOAD INTO trafodion.sch.demo SELECT * FROM hive.hive.demo;
+
+Task: LOAD             Status: Started    Object: TRAFODION.SCH.DEMO
+Task:  DISABLE INDEXE  Status: Started    Object: TRAFODION.SCH.DEMO
+Task:  DISABLE INDEXE  Status: Ended      Object: TRAFODION.SCH.DEMO
+Task:  UPSERT USING L  Status: Started    Object: TRAFODION.SCH.DEMO
+       Rows Processed: 200000 
+Task:  UPSERT USING L  Status: Ended      ET: 00:01:03.715
+Task:  POPULATE INDEX  Status: Started    Object: TRAFODION.SCH.DEMO
+Task:  POPULATE INDEX  Status: Ended      ET: 00:03:11.323
+```
+
+
+
+
+
+

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/load_transform/src/asciidoc/index.adoc
----------------------------------------------------------------------
diff --git a/docs/load_transform/src/asciidoc/index.adoc b/docs/load_transform/src/asciidoc/index.adoc
new file mode 100644
index 0000000..7f41e22
--- /dev/null
+++ b/docs/load_transform/src/asciidoc/index.adoc
@@ -0,0 +1,68 @@
+////
+* @@@ START COPYRIGHT @@@                                                         
+*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements.  See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership.  The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License.  You may obtain a copy of the License at
+*
+*   http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied.  See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*
+* @@@ END COPYRIGHT @@@ 
+////
+
+= Load and Transform Guide
+:doctype: book
+:numbered:
+:toc: left
+:toclevels: 3
+:toc-title: Table of Contents
+:icons: font
+:iconsdir: icons
+:experimental:
+:source-language: text
+:revnumber: {project-version}
+:title-logo-image: ../../../shared/trafodion-logo.jpg
+
+:images: ../images
+:leveloffset: 1
+
+// The directory is called _chapters because asciidoctor skips direct
+// processing of files found in directories starting with an _. This
+// prevents each chapter being built as its own book.
+
+**License Statement**
+
+Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements.
 See the NOTICE file
+distributed with this work for additional information regarding copyright ownership.  The
ASF licenses this file
+to you under the Apache License, Version 2.0 (the "License"); you may not use this file except
in compliance with
+the License.  You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the
License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
See the License for the
+specific language governing permissions and limitations under the License.
+
+[cols="2",options="header"]
+|===
+| Version    | Date
+| 1.3.0      | January, 2016  
+|===
+
+include::asciidoc/_chapters/about.adoc[]
+include::asciidoc/_chapters/introduction.adoc[]
+include::asciidoc/_chapters/tables_indexes.adoc[]
+include::asciidoc/_chapters/bulk_load.adoc[]
+include::asciidoc/_chapters/trickle_load.adoc[]
+include::asciidoc/_chapters/bulk_unload.adoc[]
+include::asciidoc/_chapters/monitor.adoc[]
+include::asciidoc/_chapters/troubleshoot.adoc[]

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/messages_guide/src/asciidoc/_chapters/about.adoc
----------------------------------------------------------------------
diff --git a/docs/messages_guide/src/asciidoc/_chapters/about.adoc b/docs/messages_guide/src/asciidoc/_chapters/about.adoc
index 3b4bab4..fe6c999 100644
--- a/docs/messages_guide/src/asciidoc/_chapters/about.adoc
+++ b/docs/messages_guide/src/asciidoc/_chapters/about.adoc
@@ -96,9 +96,11 @@ INTERVAL { start-field TO end-field }
 { single-field } 
 INTERVAL { start-field TO end-field | single-field }
 ``` 
+
 * | Vertical Line 
 +
 A vertical line separates alternatives in a horizontal list that is enclosed in brackets
or braces.
++
 ```
 {expression | NULL} 
 ```
@@ -116,6 +118,7 @@ ATTRIBUTE[S] attribute [, attribute] ...
 An ellipsis immediately following a single syntax item indicates that you can repeat that
syntax item any number of times.
 +
 For example:
++
 ```
 expression-n ...
 ```

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/odb_user/src/asciidoc/_chapters/about.adoc
----------------------------------------------------------------------
diff --git a/docs/odb_user/src/asciidoc/_chapters/about.adoc b/docs/odb_user/src/asciidoc/_chapters/about.adoc
index f7128a7..5e418ff 100644
--- a/docs/odb_user/src/asciidoc/_chapters/about.adoc
+++ b/docs/odb_user/src/asciidoc/_chapters/about.adoc
@@ -97,9 +97,11 @@ INTERVAL { start-field TO end-field }
 { single-field } 
 INTERVAL { start-field TO end-field | single-field }
 ``` 
+
 * | Vertical Line 
 +
 A vertical line separates alternatives in a horizontal list that is enclosed in brackets
or braces.
++
 ```
 {expression | NULL} 
 ```
@@ -117,6 +119,7 @@ ATTRIBUTE[S] attribute [, attribute] ...
 An ellipsis immediately following a single syntax item indicates that you can repeat that
syntax item any number of times.
 +
 For example:
++
 ```
 expression-n ...
 ```

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/odb_user/src/asciidoc/_chapters/load.adoc
----------------------------------------------------------------------
diff --git a/docs/odb_user/src/asciidoc/_chapters/load.adoc b/docs/odb_user/src/asciidoc/_chapters/load.adoc
index 3d6a1ee..b735b7a 100644
--- a/docs/odb_user/src/asciidoc/_chapters/load.adoc
+++ b/docs/odb_user/src/asciidoc/_chapters/load.adoc
@@ -40,7 +40,8 @@ This command:
 * in the table `TRAFODION.MAURIZIO.CUSTOMER` (`tgt=TRAFODION.MAURIZIO.CUSTOMER`)
 * using `|` (vertical bar) as a field separator (`fs=\|`)
 * using `1000 rows` as row-set buffer (`rows=1000`)
-* using `UPSERT USING LOAD` syntax to achieve better throughput as described in http://trafodion.incubator.apache.org/load.html[Trafodion
Data Loading]
+* using `UPSERT USING LOAD` syntax to achieve better throughput as described in
+http://trafodion.incubator.apache.org/docs/load_transform/index.html[Trafodion Load and Transform
Guide]
 * truncating the target table before loading (`truncate`)
 * using `4 parallel threads` to load the target table (`parallel=4`)
 
@@ -382,37 +383,37 @@ The following example explains mapfile usage to skip/transform or generate
field
 
 And an input file like this:
 
-```
-uno,00,51,due,_Maurizio_,tre,[underline]#07 Mar 1959#, ignore,remaining, fields
-uno,00,46,due,_Lucia_,tre,[underline]#13 Oct 1964#, ignore, this
-uno,00,34,due,_Giovanni_,tre,[underline]#30 Mar 1976#
-uno,00,48,due,_Antonella_,tre,[underline]#24 Apr 1962#
-```
+***
+uno,00,*51*,due,_Maurizio_,tre,[underline]#07 Mar 1959#, ignore,remaining, fields +
+uno,00,*46*,due,_Lucia_,tre,[underline]#13 Oct 1964#, ignore, this +
+uno,00,*34*,due,_Giovanni_,tre,[underline]#30 Mar 1976# +
+uno,00,*48*,due,_Antonella_,tre,[underline]#24 Apr 1962#
+***
 
 * *Bold text* represents age.
 * _Italics  text_ represents name.
 * [underline]#Underline text# represents birth date.
 
-You want to load the marked fields into the appropriate column, *_generate_* a unique key
for ID and ignore the fields in black text.
+You want to load the marked fields into the appropriate column, *_generate_* a unique key
for ID and ignore the remaining fields.
 In addition: you need to *_convert date format_* and replace all occurrences of `Lucia` with
`Lucy`.
 
 The following map file accomplishes these goals:
 
-```
-~/Devel/odb $ cat test/load_map/ml1.map
-# Map file to load TRAFODION.MFTEST.FRIENDS from friends.dat
-ID:seq:1                  <- Inserts into ID column a sequence starting from 1
-NAME:4:REPLACE:Lucia:Lucy <- Loads field #4 into NAME and replace all occurrences of Lucia
with Lucy
-AGE:2                     <- Loads field #2 (they start from zero) into AGE 
-BDATE:6:DCONV:d.b.y       <- Loads field #6 into BDATE converting date format from dd
mmm yyyy
-```
+***
+$ cat test/load_map/ml1.map +
+# Map file to load TRAFODION.MFTEST.FRIENDS from friends.dat +
+ID:seq:1                  # Inserts into ID column a sequence starting from 1 +
+NAME:4:REPLACE:Lucia:Lucy # Loads field #4 into NAME and replace all occurrences of Lucia
with Lucy +
+AGE:2                     # Loads field #2 (they start from zero) into AGE +
+BDATE:6:DCONV:d.b.y       # Loads field #6 into BDATE converting date format from dd mmm
yyyy
+***
 
 <<<
 Load as follows:
 
 ```
 $ odb64luo –u user –p xx –d dsn \
-  -l src=frends.dat:tgt=TRAFODION.MFTEST.FRIENDS:map=ml1.map:fs=,
+  -l src=friends.dat:tgt=TRAFODION.MFTEST.FRIENDS:map=ml1.map:fs=,
 ```
 
 == Use mapfiles to Load Fixed Format Files
@@ -572,11 +573,11 @@ create table TRAFODION.maurizio.dtest
 
 If you have an input file containing:
 
-```
-ignoreme,Maurizio,xyz,_commentM_, ignore,remaining, fields
-ignoreme,Lucia,xyz,_commentL_, ignore, this
-ignoreme,Giovanni,xyz,_commentG_, ignoreme,Antonella,xyz,commentA
-```
+***
+ignoreme,*Maurizio*,xyz,_commentM_, ignore,remaining, fields
+ignoreme,*Lucia*,xyz,_commentL_, ignore, this
+ignoreme,*Giovanni*,xyz,_commentG_, ignoreme,Antonella,xyz,commentA
+***
 
 * *Bold text* represents `fname`.
 * _Italic text_ represents `comment`.
@@ -759,7 +760,7 @@ The example above:
 * Saves data into files `ext_%t.csv.gz` (`%t` is expanded to the real table name).
 * Compresses the output file (gzip) on the fly (uncompressed data never lands to disk).
 * Trims text fields.
-* Used a 10 MB IO buffer.
+* Uses a 10 MB IO buffer.
 * Uses three threads (ODBC connection) for the extraction process.
 
 === Extraction Options
@@ -1161,7 +1162,6 @@ src=TRAFODION.MAURIZIO.PART:splitby=P_PARTKEY
 src=TRAFODION.MAURIZIO.LINEITEM:splitby=L_PARTKEY
 ```
 
-
 == Case-Sensitive Table and Column Names
 
 Your database configuration determines whether you can use case sensitive table/column names.

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/provisioning_guide/src/asciidoc/_chapters/about.adoc
----------------------------------------------------------------------
diff --git a/docs/provisioning_guide/src/asciidoc/_chapters/about.adoc b/docs/provisioning_guide/src/asciidoc/_chapters/about.adoc
index 1cd4b2f..b5a67e9 100644
--- a/docs/provisioning_guide/src/asciidoc/_chapters/about.adoc
+++ b/docs/provisioning_guide/src/asciidoc/_chapters/about.adoc
@@ -117,6 +117,7 @@ INTERVAL { start-field TO end-field | single-field }
 * | Vertical Line 
 +
 A vertical line separates alternatives in a horizontal list that is enclosed in brackets
or braces.
++
 ```
 {expression | NULL} 
 ```
@@ -133,6 +134,7 @@ ATTRIBUTE[S] attribute [, attribute] ...
 An ellipsis immediately following a single syntax item indicates that you can repeat that
syntax item any number of times.
 +
 For example:
++
 ```
 expression-n ...
 ```

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/sql_reference/src/asciidoc/_chapters/about.adoc
----------------------------------------------------------------------
diff --git a/docs/sql_reference/src/asciidoc/_chapters/about.adoc b/docs/sql_reference/src/asciidoc/_chapters/about.adoc
index fdb1ffe..de3dcb8 100644
--- a/docs/sql_reference/src/asciidoc/_chapters/about.adoc
+++ b/docs/sql_reference/src/asciidoc/_chapters/about.adoc
@@ -155,6 +155,7 @@ INTERVAL { start-field TO end-field | single-field }
 * | Vertical Line 
 +
 A vertical line separates alternatives in a horizontal list that is enclosed in brackets
or braces.
++
 ```
 {expression | NULL} 
 ```
@@ -170,6 +171,7 @@ ATTRIBUTE[S] attribute [, attribute] ...
 An ellipsis immediately following a single syntax item indicates that you can repeat that
syntax item any number of times.
 +
 For example:
++
 ```
 expression-n ...
 ```

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc
----------------------------------------------------------------------
diff --git a/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc b/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc
index 26f0e7b..c96d125 100644
--- a/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc
+++ b/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc
@@ -31,26 +31,6 @@
 This section describes CQDs that are used to override system-level
 default settings.
 
-[[hbase_environment_cqds]]
-== HBase Environment CQDs
-
-Defines the HBase interface.
-
-[[cqd_hbase_interface]]
-=== HBASE_INTERFACE
-
-[cols="25%h,75%"]
-|===
-| Category                    | HBase
-| Description                 | Interface to use to access HBase. 
-| Values                      | Specify one of these values: +
- +
-- JNI to use a JNI interface +
-- JNI_TRX to use a transactional interface with HBase-trx via JNI. +
- +
- The default value is JNI_TRX.
-|===
-
 [[hive_environment_cqds]]
 == Hive Environment CQDs
 

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc
----------------------------------------------------------------------
diff --git a/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc b/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc
index 45315c5..80ce893 100644
--- a/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc
+++ b/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc
@@ -1123,16 +1123,16 @@ EXECUTE y;                              -- uses MYSCHEMA;
 [[control_query_default_examples]]
 === Examples of CONTROL QUERY DEFAULT
 
-* Change the maximum supported length of the column names to 200 for the current process:
+* Increase the cache refresh time for the histogram cache to two hours (7,200 minutes).
 +
 ```
-CONTROL QUERY DEFAULT HBASE_MAX_COLUMN_NAME_LENGTH '200';
+CONTROL QUERY DEFAULT CACHE_HISTOGRAMS_REFRESH_INTERVAL '7200';
 ```
 
-* Reset the HBASE_MAX_COLUMN_NAME_LENGTH attribute to its initial value in the current process:
+* Reset the CACHE_HISTOGRAMS_REFRESH_INTERVAL attribute to its initial value in the current
process:
 +
 ```
-CONTROL QUERY DEFAULT HBASE_MAX_COLUMN_NAME_LENGTH RESET;
+CONTROL QUERY DEFAULT CACHE_HISTOGRAMS_REFRESH_INTERVAL RESET;
 ```
 
 <<<

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/sql_reference/src/asciidoc/_chapters/sql_utilities.adoc
----------------------------------------------------------------------
diff --git a/docs/sql_reference/src/asciidoc/_chapters/sql_utilities.adoc b/docs/sql_reference/src/asciidoc/_chapters/sql_utilities.adoc
index 2e0060b..4de3903 100644
--- a/docs/sql_reference/src/asciidoc/_chapters/sql_utilities.adoc
+++ b/docs/sql_reference/src/asciidoc/_chapters/sql_utilities.adoc
@@ -59,16 +59,26 @@ associated with it. The write path begins at a client, moves to a region
 server, and ends when data eventually is written to an HBase data file
 called an HFile.
 
-The Trafodion bulk load process takes place in two phases:
+The Trafodion bulk load process takes place in the following phases:
 
-* _Preparation phase_: In this phase, Trafodion reads the data from
-the source files in Hive or HDFS, partitions the data based on the
-target table's partitioning scheme, sorts the data, and then generates
-key-value pairs that will populate the HFiles. Trafodion also encodes the
-data for faster storage and retrieval.
-* _Loading-the-files-into-HBase phase_: This phase uses the
-LoadIncrementalHFiles (also known as the computebulkload tool) and load
-the generated HFiles into the region servers.
+* *Disable Indexes* (if incremental index build not used)
+
+* *Prepare* (takes most time, heart of the bulk load operation)
+** Read source files (Trafodion Table, Hive table, or Hive external table)
+** Data encoded in Trafodion encoding
+** Data repartitioned and sorted to match regions of target table
+** Data written to HFiles
+** Data repartitioned and written to index HFiles (if incremental index build IS used)
+
+* *Complete* (with or without Snapshot recovery)
+** Take a snapshot of the table
+** Merge HFiles into HBase table (very fast – move, not a copy)
+** Delete snapshot or restore from snapshot if merge fails
+
+* *Populate Indexes* (if incremental index build is NOT used)
+
+* *Cleanup*
+** HFiles temporary space cleanup
 
 LOAD is a Trafodion SQL extension.
 
@@ -121,6 +131,7 @@ specifies that the Bulk Loader not use HBase snapshots for recovery. By
 default, the Bulk Loader handles recovery using the HBase snapshots
 mechanism.
 
+<<<
 ** `NO POPULATE INDEXES`
 +
 specifies that the Bulk Loader not handle index maintenance or populate
@@ -176,10 +187,10 @@ guidelines.
 ==== Staging Folder for HFiles
 
 The Bulk Loader uses an HDFS folder as a staging area for the HFiles
-before calling HBase APIs to merge them into the Trafodion table. By
-default, Trafodion uses /bulkload/ as the staging
-folder. This folder must be owned by the same user as the one under
-which Trafodion runs. Trafodion
+before calling HBase APIs to merge them into the Trafodion table.
+
+By default, Trafodion uses /bulkload as the staging folder. This folder
+must be owned by the same user as the one under which Trafodion runs. Trafodion
 also must have full permissions on this folder. The HBase user (that is,
 the user under which HBase runs) must have read/write access to this
 folder.
@@ -190,27 +201,99 @@ Example:
 drwxr-xr-x - trafodion trafodion 0 2014-07-07 09:49 /bulkload.
 ```
 
+<<<
+==== Improving Throughput
+
+The following CQD (Control Query Default) settings help improve the Bulk Loader
+throughput:
+
+* `TRAF_LOAD_MAX_HFILE_SIZE`
++
+Specifies the HFile size limit beyond which the current file is closed and a
+new file is created for the same partition. Adjust this setting to minimize
+HBase splitting/merging.
++
+*Default*: 10GB
+
+* `TRAF_LOAD_PREP_TMP_LOCATION`
++
+Specifies the HDFS directory where HFiles are created during load.
++
+*Default*: `/bulkload`
+
+Also, consider using `NO DUPLICATE CHECK` to improve througput if your
+source data is clean.
+
 ==== Hive Source Table
 
-To load data stored in HDFS, you will need to create a Hive table with
+To load data stored in HDFS, you need to create a Hive table with
 the right fields and types pointing to the HDFS folder containing the
 data before you start the load.
 
-<<<
 ==== HBase Snapshots
 
 If you do not specify the NO RECOVERY OPTION in the LOAD statement, the
 Bulk Loader uses HBase snapshots as a mechanism for recovery. Snapshots
 are a lightweight operation where some metadata is copied. (Data is not
-copied.) A snapshot is taken before the load starts and is removed after
+copied.)
+
+A snapshot is taken before the load starts and is removed after
 the load completes successfully. If something goes wrong and it is
 possible to recover, the snapshot is used to restore the table to its
 initial state before the load started. To use this recovery mechanism,
 HBase needs to be configured to allow snapshots.
 
+==== Incremental Loads
+
+The Bulk Loader allows for incremental loads by default. Snapshots are
+taken before second phase starts and deleted once the bulk load completes.
+
+If something goes wrong with the load, then the snapshot is restored to
+go to the previous state.
+
+<<<
+==== Non-Incremental Loads
+
+These following bulk load options can be used to do non-incremental load:
+
+* `NO RECOVERY`: Do not take a snapshot of the table.
+* `TRUNCATE TABLE`: Truncates the table before starting the load.
+
+==== Space Usage
+
+The target table values for SYSKEY, SALT, identity, divisioning columns
+are created automatically the during transformation step. The size of the
+HBase files is determined based on encoding, compression, HDFS replication
+factor, and row format. Target table can be pre-split into regions using
+salting, a Java Program, by seeding the table with data.
+
+==== Performance
+
+The overall throughput is influenced by row format, row length, number of
+columns, skew in data, etc. LOAD sas upsert semantics (duplicate constraint
+not checked with existing data). LOAD has lower CPU abd disk activity than
+similar trickle load (INSERT, UPSERT, or UPSERT USING LOAD), Also, LOAD has
+lower compaction activity after completion than Trickle Load. 
+
+==== Hive Scans
+
+Direct access for Hive table data supports:
+
+* Only text input format and sequence files.
+* Only structured data types.
+
+Tables must be created/dropped/altered through Hive itself.
+
+Trafodion:
+
+* Reads Hive metadata to determine information about table.
+* UPDATE STATISTICS can be performed on Hive tables - improves performance! 
+* Can write to Hive tables in both Text and Sequence formats (used by UNLOAD).
+
+<<<
 [[load_examples]]
 === Examples of LOAD
-
+    
 * For customer demographics data residing in
 `/hive/tpcds/customer_demographics`, create an external Hive table using
 the following Hive SQL:

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/markdown/download.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/download.md b/docs/src/site/markdown/download.md
index 55eebe3..66e0c2f 100644
--- a/docs/src/site/markdown/download.md
+++ b/docs/src/site/markdown/download.md
@@ -121,22 +121,8 @@ cd $HOME/trafodion/installer
 ./trafodion_install --accept_license --config_file my_config
 ```
 
-The Trafodion Installer performs all the operations required to install and start Trafodion.
Type **`initialize trafodion;`** 
-when the **`trafci`** prompt (**`>>`**) appears.
-
-**Example**
-
-```
-Apache Trafodion Conversational Interface 1.3.0
-Copyright (c) 2015 Apache Software Foundation
->> initialize trafodion;
---- SQL operation complete.
->>
-
-End of MXCI Session
-
-***INFO: Installation completed successfully.
-```
+The Trafodion Installer performs all the operations required to install and start Trafodion.
Wait for it to complete before
+continuing to the next step.
 
 ## Verify Installation
 

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/index.md b/docs/src/site/markdown/index.md
index fc844cb..8450368 100644
--- a/docs/src/site/markdown/index.md
+++ b/docs/src/site/markdown/index.md
@@ -46,11 +46,11 @@ Trafodion provides SQL access to structured, semi-structured, and unstructured
d
   <p><h5>The Apache Trafodion team is happy to announce the first Apache release
&#8212; version 1.3.0-incubating!</h5></p>
   <p><strong>How to get started with Apache Trafodion:</strong>
      <ul>
-       <li><a href="http://trafodion.apache.org/documentation.html">Release Documentation</a></li>
-       <li><a href="http://trafodion.apache.org/release-notes-1-3-0.html">Release
Notes</a></li>
-       <li><a href="https://trafodion.apache.org/download.html#Binaries">Download</a></li>
+       <li><a href="http://trafodion.incubator.apache.org/documentation.html">Release
Documentation</a></li>
+       <li><a href="http://trafodion.incubator.apache.org/release-notes-1-3-0.html">Release
Notes</a></li>
+       <li><a href="https://trafodion.incubator.apache.org/download.html">Download</a></li>
        <li><a href="https://github.com/apache/incubator-trafodion">git Site</a></li>
-       <li><a href="http://trafodion.apache.org/download.html">Build Instructions</a></li>
+       <li><a href="http://trafodion.incubator.apache.org/contributing-redirect.html">Build
Instructions</a></li>
      </ul>
    </p>
    <p><h5>The Trafodion team thanks all our contributors and mentors who made
the release possible!</h5></p>
@@ -126,9 +126,9 @@ Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Traf
         <div class="customHr">.</div>
       </center>
       <ul>
-        <li><a href="contribution-redirect.html">Contribute</a></li>
+        <li><a href="contributing-redirect.html">Contribute</a></li>
         <li><a href="mail-lists.html">Discuss</a></li>
-        <li><a href="calendar.html">Calendar</a></li>
+        <li><a href="https://cwiki.apache.org/confluence/display/TRAFODION/Apache+Trafodion+Home">Wiki</a></li>
         <li><a href="presentations.html">Presentations</a></li>
       </ul>
     </td>

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/markdown/logo.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/logo.md b/docs/src/site/markdown/logo.md
new file mode 100644
index 0000000..f621a28
--- /dev/null
+++ b/docs/src/site/markdown/logo.md
@@ -0,0 +1,56 @@
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+ 
+      http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the 
+  License.
+-->
+
+You may use the Trafodion logo on social media, presentations, and other materials
+when referencing our project. Download as needed.
+
+---
+
+#### [trafodion-dragon-large.jpg][1]
+
+[![Trafodion Dragon Large jpg](images/logos/trafodion-dragon-large.jpg)][1]
+
+[1]: images/logos/trafodion-dragon-large.png
+
+#### [trafodion-dragon-large.png][2]
+
+[![Trafodion Dragon Large png](images/logos/trafodion-dragon-large.png)][2]
+
+[2]: images/logos/trafodion-dragon-large.png
+
+#### [pb-trafodion.jpg][5]
+
+[![Powered by Trafodion jpg](images/logos/pb-trafodion.jpg)][5]
+
+[5]: images/logos/pb-trafodion.jpg
+
+#### [pb-trafodion.png][6]
+
+[![Powered by Trafodion png](images/logos/pb-trafodion.png)][5]
+
+[6]: images/logos/pb-trafodion.png
+
+#### [trafodion-logo.jpg][3]
+
+[![Trafodion Logo jog](images/logos/trafodion-logo.jpg)][3]
+
+[3]: images/logos/trafodion-logo.jpg
+
+#### [trafodion-dragon.png][4]
+
+[![Trafodion Dragon png](images/logos/trafodion-dragon.png)][4]
+
+[4]: images/logos/trafodion-dragon.png
+

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/css/site.css
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/css/site.css b/docs/src/site/resources/css/site.css
index 72806f2..c415896 100644
--- a/docs/src/site/resources/css/site.css
+++ b/docs/src/site/resources/css/site.css
@@ -20,7 +20,90 @@
 *
 * @@@ END COPYRIGHT @@@ 
  */
- 
+@import url(https://fonts.googleapis.com/css?family=Oswald:400,300);
+
+/* navbar */
+.navbar-default {
+    background-color: #F8F8F8;
+    border-color: #E7E7E7;
+}
+/* title */
+.navbar-default .navbar-brand {
+    color: #777;
+}
+.navbar-default .navbar-brand:hover,
+.navbar-default .navbar-brand:focus {
+    color: #5E5E5E;
+}
+/* link */
+.navbar-default .navbar-nav > li > a {
+    color: #777;
+}
+.navbar-default .navbar-nav > li > a:hover,
+.navbar-default .navbar-nav > li > a:focus {
+    color: #333;
+}
+.navbar-default .navbar-nav > .active > a, 
+.navbar-default .navbar-nav > .active > a:hover, 
+.navbar-default .navbar-nav > .active > a:focus {
+    color: #555;
+    background-color: #E7E7E7;
+}
+.navbar-default .navbar-nav > .open > a, 
+.navbar-default .navbar-nav > .open > a:hover, 
+.navbar-default .navbar-nav > .open > a:focus {
+    color: #555;
+    background-color: #D5D5D5;
+}
+/* caret */
+.navbar-default .navbar-nav > .dropdown > a .caret {
+    border-top-color: #777;
+    border-bottom-color: #777;
+}
+.navbar-default .navbar-nav > .dropdown > a:hover .caret,
+.navbar-default .navbar-nav > .dropdown > a:focus .caret {
+    border-top-color: #333;
+    border-bottom-color: #333;
+}
+.navbar-default .navbar-nav > .open > a .caret, 
+.navbar-default .navbar-nav > .open > a:hover .caret, 
+.navbar-default .navbar-nav > .open > a:focus .caret {
+    border-top-color: #555;
+    border-bottom-color: #555;
+}
+/* mobile version */
+.navbar-default .navbar-toggle {
+    border-color: #DDD;
+}
+.navbar-default .navbar-toggle:hover,
+.navbar-default .navbar-toggle:focus {
+    background-color: #DDD;
+}
+.navbar-default .navbar-toggle .icon-bar {
+    background-color: #CCC;
+}
+@media (max-width: 767px) {
+    .navbar-default .navbar-nav .open .dropdown-menu > li > a {
+        color: #777;
+    }
+    .navbar-default .navbar-nav .open .dropdown-menu > li > a:hover,
+    .navbar-default .navbar-nav .open .dropdown-menu > li > a:focus {
+        color: #333;
+    }
+}
+
+.brand-apache {
+        color: white;
+        font-family: 'Oswald',sans-serif; font-weight; 300;
+        font-size: 20px;
+}
+
+.brand-trafodion {
+        color: white;
+        font-family: 'Oswald',sans-serif; font-weight; 400;
+        font-size: 30px;
+}
+
 .color-highlight {
 	color: red;
 }
@@ -97,7 +180,6 @@ h3, h4, h5, h6 {
   margin-top: 10px;
 }
 
-
 @media (min-width: 980px) {
   .page-themes-bootswatch-readable .navbar.affix {
     top: 60px;
@@ -110,5 +192,7 @@ h3, h4, h5, h6 {
   .page-themes-bootswatch-spruce .navbar.affix {
     top: 55px;
   }
+
 }
 
+

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/carousel/scale.png
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/carousel/scale.png b/docs/src/site/resources/images/carousel/scale.png
index 58f488d..af9651d 100644
Binary files a/docs/src/site/resources/images/carousel/scale.png and b/docs/src/site/resources/images/carousel/scale.png
differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/carousel/stack.png
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/carousel/stack.png b/docs/src/site/resources/images/carousel/stack.png
index 7cb7673..be31b8a 100644
Binary files a/docs/src/site/resources/images/carousel/stack.png and b/docs/src/site/resources/images/carousel/stack.png
differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/logos/pb-Trafodion.png
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/logos/pb-Trafodion.png b/docs/src/site/resources/images/logos/pb-Trafodion.png
new file mode 100644
index 0000000..67643a4
Binary files /dev/null and b/docs/src/site/resources/images/logos/pb-Trafodion.png differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/logos/pb-trafodion.jpg
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/logos/pb-trafodion.jpg b/docs/src/site/resources/images/logos/pb-trafodion.jpg
new file mode 100644
index 0000000..0042df9
Binary files /dev/null and b/docs/src/site/resources/images/logos/pb-trafodion.jpg differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/logos/trafodion-dragon-large.jpg
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/logos/trafodion-dragon-large.jpg b/docs/src/site/resources/images/logos/trafodion-dragon-large.jpg
new file mode 100644
index 0000000..2380059
Binary files /dev/null and b/docs/src/site/resources/images/logos/trafodion-dragon-large.jpg
differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/logos/trafodion-dragon-large.png
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/logos/trafodion-dragon-large.png b/docs/src/site/resources/images/logos/trafodion-dragon-large.png
new file mode 100644
index 0000000..2d75319
Binary files /dev/null and b/docs/src/site/resources/images/logos/trafodion-dragon-large.png
differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/logos/trafodion-dragon.png
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/logos/trafodion-dragon.png b/docs/src/site/resources/images/logos/trafodion-dragon.png
new file mode 100644
index 0000000..58973e8
Binary files /dev/null and b/docs/src/site/resources/images/logos/trafodion-dragon.png differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/resources/images/logos/trafodion-logo.jpg
----------------------------------------------------------------------
diff --git a/docs/src/site/resources/images/logos/trafodion-logo.jpg b/docs/src/site/resources/images/logos/trafodion-logo.jpg
index b1a1796..8d5c730 100644
Binary files a/docs/src/site/resources/images/logos/trafodion-logo.jpg and b/docs/src/site/resources/images/logos/trafodion-logo.jpg
differ

http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/33d796d6/docs/src/site/site.xml
----------------------------------------------------------------------
diff --git a/docs/src/site/site.xml b/docs/src/site/site.xml
index 01fb82a..cf130d6 100644
--- a/docs/src/site/site.xml
+++ b/docs/src/site/site.xml
@@ -34,7 +34,7 @@
   <bannerLeft>
     <!-- Apache Trafodion, but with "Trafodion" highlighted -->
     <name>
-       <![CDATA[<img src="images/logos/trafodion-logo.jpg" alt="Trafodion Logo">]]>
+       <![CDATA[<img src="images/logos/trafodion-logo.jpg" alt="Trafodion Logo" width="383">]]>
     </name>
     <href>index.html</href>
   </bannerLeft>
@@ -46,14 +46,16 @@
       <theme>bootswatch-cerulean</theme>
       <titleTemplate>%1$s %2$s</titleTemplate>
       <skinAttribution>false</skinAttribution>
-      <breadcrumbs>true</breadcrumbs>   
+      <breadcrumbs>false</breadcrumbs>   
+      <imgLightbox>false</imgLightbox>
+      <markPageHeader>false</markPageHeader>
       <smoothScroll>true</smoothScroll>
       <!--  Enable code highlighting -->
       <highlightJs>true</highlightJs>
       <brand>
         <!-- Brand text in top-left part of the site -->
         <name>
-          <![CDATA[Apache <span class="color-highlight">Trafodion</span>]]>
+          <![CDATA[<span class="brand-apache">Apache</span> <span class="brand-trafodion">Trafodion</span>]]>
         </name>
         <href>index.html</href>
       </brand>
@@ -116,6 +118,14 @@
           <shortTitle>License</shortTitle>
           <toc>false</toc>
         </license>
+        <logo>
+          <shortTitle>Logo</shortTitle>
+          <toc>false</toc>
+           <sections>
+              <body />
+              <thumbs>2</thumbs>
+           </sections>
+        </logo>        
         <mail-lists>
           <shortTitle>Mailing Lists</shortTitle>
           <toc>false</toc>
@@ -188,13 +198,14 @@
     <links>
       <item name="Download" href="download.html"/>
       <item name="Quick Start" href="quickstart.html"/>
-      <item name="Docs" href="documentation.html"/>
+      <item name="Documentation" href="documentation.html"/>
       <item name="Contribute" href="contributing-redirect.html"/>
     </links>
     <menu name="Project">
       <item href="index.html" name="Overview"/>
       <item href="faq.html" name="FAQ"/>
       <item href="release-notes.html" name="Release Notes"/>
+      <item href="https://cwiki.apache.org/confluence/display/TRAFODION/Apache+Trafodion+Home"
name="Wiki"/>
       <item href="management.html" name="Managing Trafodion"/>
       <!-- item href="new-features.html" name="New Features"/-->
       <!-- item href="roadmap.html" name="Roadmap"/-->
@@ -202,6 +213,7 @@
       <item href="http:divider" name=""/>
       <item href="team-redirect.html" name="Team"/>
       <item href="presentations.html" name="Presentations"/>
+      <item href="logo.html" name="Logo"/>
       <item href="mail-lists.html" name="Mailing List"/>
       <item href="http:divider" name=""/>
       <item href="source-repository.html" name="Source Repository"/>



Mime
View raw message