carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jack...@apache.org
Subject [46/54] [abbrv] carbondata git commit: [CARBONDATA-1117]updated configuration-parameters.md and useful-tips-on-carbondata.md files for SET/RESET
Date Thu, 14 Sep 2017 09:20:39 GMT
[CARBONDATA-1117]updated configuration-parameters.md and useful-tips-on-carbondata.md files
for SET/RESET

This closes #1219


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/b414393b
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/b414393b
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/b414393b

Branch: refs/heads/streaming_ingest
Commit: b414393bed319ea970036aac72f140b9389bccf9
Parents: 2176a2f
Author: vandana <vandana.yadav759@gmail.com>
Authored: Mon Jul 31 19:32:58 2017 +0530
Committer: Jacky Li <jacky.likun@qq.com>
Committed: Wed Sep 13 19:24:35 2017 +0800

----------------------------------------------------------------------
 docs/configuration-parameters.md  | 87 +++++++++++++++++++++++++++++++++-
 docs/useful-tips-on-carbondata.md |  1 -
 2 files changed, 85 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/carbondata/blob/b414393b/docs/configuration-parameters.md
----------------------------------------------------------------------
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index bdd551a..8101aa1 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -24,6 +24,7 @@
  * [Performance Configuration](#performance-configuration)
  * [Miscellaneous Configuration](#miscellaneous-configuration)
  * [Spark Configuration](#spark-configuration)
+ * [Dynamic Configuration In CarbonData Using SET-RESET](#dynamic-configuration-in-carbondata-using-set-reset)
  
  
 ##  System Configuration
@@ -146,5 +147,87 @@ This section provides the details of all the configurations required
for CarbonD
 |----------------------------------------|--------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | spark.driver.memory | 1g | Amount of memory to be used by the driver process. |
 | spark.executor.memory | 1g | Amount of memory to be used per executor process. |
-   
- 
+
+##  Dynamic Configuration In CarbonData Using SET-RESET
+
+**SET/RESET** commands are used to add, update, display, or reset the carbondata properties
dynamically without restarting the driver.
+
+**Syntax**
+
+* **Add or Update :** This command adds or updates the value of parameter_name.
+
+```
+SET parameter_name=parameter_value
+```
+
+* Display Property Value: This command displays the value of the specified parameter_name.
+
+```
+SET parameter_name
+```
+
+* Display Session Parameters: This command displays all the supported session parameters.
+
+```
+SET
+```
+
+* Display Session Parameters along with usage details: This command displays all the supported
session parameters along with their usage details.
+
+```
+SET -v
+```
+
+* Reset: This command clears all the session parameters.
+
+```
+RESET
+```
+
+ **Parameter Description:**
+
+| Parameter       | Description                                                         
                  |
+|-----------------|----------------------------------------------------------------------------------------|
+| parameter_name  | Name of the property whose value needs to be dynamically added, updated,
or displayed. |
+| parameter_value | New value of the parameter_name to be set.                          
                  |
+
+<b><p align="center">Dynamically Configurable Properties of CarbonData</p></b>
+
+| Properties                               | Description                                
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                           |
+|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| carbon.options.bad.records.logger.enable | To enable or disable bad record logger.    
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                           |
+| carbon.options.bad.records.action        | This property can have four types of actions
for bad records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the
data by storing the bad records as NULL. If set to REDIRECT then bad records are written to
the raw CSV instead of being loaded. If set to IGNORE then bad records are neither loaded
nor written to the raw CSV. If set to FAIL then data loading fails if any bad records are
found.                                                                                   
                                                                      |
+| carbon.options.is.empty.data.bad.record  | If false, then empty ("" or '' or ,,) data will
not be considered as bad record and vice versa.                                          
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                       |
+| carbon.options.sort.scope                | This property can have four possible values
BATCH_SORT, LOCAL_SORT, GLOBAL_SORT and NO_SORT. If set to BATCH_SORT, the sorting scope is
smaller and more index tree will be created,thus loading is faster but query maybe slower.
If set to LOCAL_SORT, the sorting scope is bigger and one index tree per data node will be
created, thus loading is slower but query is faster. If set to GLOBAL_SORT, the sorting scope
is bigger and one index tree per task will be created, thus loading is slower but query is
faster. If set to NO_SORT data will be loaded in unsorted manner. |
+| carbon.options.batch.sort.size.inmb      | Size of batch data to keep in memory, as a thumb
rule it supposed to be less than 45% of sort.inmemory.size.inmb otherwise it may spill intermediate
data to disk.                                                                            
                                                                                         
                                                                                         
                                                                                         
                                                            |
+| carbon.options.single.pass               | Single Pass Loading enables single job to finish
data loading with dictionary generation on the fly. It enhances performance in the scenarios
where the subsequent data loading after initial load involves fewer incremental updates on
the dictionary. This option specifies whether to use single pass for loading data or not.
By default this option is set to FALSE.                                                  
                                                                                         
                                                                  |
+| carbon.options.bad.record.path           | Specifies the HDFS path where bad records needs
to be stored.                                                                            
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                       |
+| carbon.options.global.sort.partitions    | The Number of partitions to use when shuffling
data for sort. If user don't configurate or configurate it less than 1, it uses the number
of map tasks as reduce tasks. In general, we recommend 2-3 tasks per CPU core in your cluster.
                                                                                         
                                                                                         
                                                                                         
                                                                  |
+| carbon.custom.block.distribution         | Specifies whether to use the Spark or Carbon
block distribution feature.                                                              
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                          |
+| enable.unsafe.sort                       | Specifies whether to use unsafe sort during
data loading. Unsafe sort reduces the garbage collection during data load operation, resulting
in better performance.                                                                   
                                                                                         
                                                                                         
                                                                                         
                                                                      |
+
+**Examples:**
+
+* Add or Update:
+
+```
+SET enable.unsafe.sort =true
+```
+
+* Display Property Value:
+
+```
+SET enable.unsafe.sort
+```
+
+* Reset:
+
+```
+RESET
+```
+
+**System Response:**
+
+* Success will be recorded in the driver log.
+
+* Failure will be displayed in the UI.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/carbondata/blob/b414393b/docs/useful-tips-on-carbondata.md
----------------------------------------------------------------------
diff --git a/docs/useful-tips-on-carbondata.md b/docs/useful-tips-on-carbondata.md
index 6c73b5e..d1d4a8c 100644
--- a/docs/useful-tips-on-carbondata.md
+++ b/docs/useful-tips-on-carbondata.md
@@ -25,7 +25,6 @@ The following sections will elaborate on the above topics :
 * [Configuration for Optimizing Data Loading performance for Massive Data](#configuration-for-optimizing-data-loading-performance-for-massive-data)
 * [Optimizing Mass Data Loading](#configurations-for-optimizing-carbondata-performance)
 
-
 ## Suggestions to Create CarbonData Table
 
 Recently CarbonData was used to analyze performance of Telecommunication field.


Mime
View raw message