carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jack...@apache.org
Subject carbondata git commit: [CARBONDATA-1536] Default value of carbon.bad.records.action is FORCE
Date Sun, 01 Oct 2017 12:43:23 GMT
Repository: carbondata
Updated Branches:
  refs/heads/master 349c59c7b -> 7d3bf0503


[CARBONDATA-1536] Default value of carbon.bad.records.action is FORCE

Default value of carbon.bad.records.action is 'FORCE' in source code, but the value is 'FAIL'
in documentation. This PR fix documentation

This closes #1396


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/7d3bf050
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/7d3bf050
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/7d3bf050

Branch: refs/heads/master
Commit: 7d3bf05030e4f3112aa487438307d3e93bf950c5
Parents: 349c59c
Author: xubo245 <601450868@qq.com>
Authored: Thu Sep 28 23:15:21 2017 +0800
Committer: Jacky Li <jacky.likun@qq.com>
Committed: Sun Oct 1 20:43:04 2017 +0800

----------------------------------------------------------------------
 docs/configuration-parameters.md                               | 2 +-
 docs/dml-operation-on-carbondata.md                            | 2 +-
 .../spark/testsuite/dataload/TestBatchSortDataLoad.scala       | 6 +-----
 3 files changed, 3 insertions(+), 7 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/carbondata/blob/7d3bf050/docs/configuration-parameters.md
----------------------------------------------------------------------
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index 77320f6..e085317 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -60,7 +60,7 @@ This section provides the details of all the configurations required for
CarbonD
 | carbon.update.persist.enable | true | Enabling this parameter considers persistent data.
Enabling this will reduce the execution time of UPDATE operation. |  |
 | carbon.load.global.sort.partitions | 0 | The Number of partitions to use when shuffling
data for sort. If user don't configurate or configurate it less than 1, it uses the number
of map tasks as reduce tasks. In general, we recommend 2-3 tasks per CPU core in your cluster.
 | carbon.options.bad.records.logger.enable | false | Whether to create logs with details
about bad records. | |
-| carbon.bad.records.action | fail | This property can have four types of actions for bad
records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by
storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw
CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written
to the raw CSV. If set to FAIL then data loading fails if any bad records are found. | |
+| carbon.bad.records.action | FORCE | This property can have four types of actions for bad
records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by
storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw
CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written
to the raw CSV. If set to FAIL then data loading fails if any bad records are found. | |
 | carbon.options.is.empty.data.bad.record | false | If false, then empty ("" or '' or ,,)
data will not be considered as bad record and vice versa. | |
 | carbon.options.bad.record.path |  | Specifies the HDFS path where bad records are stored.
By default the value is Null. This path must to be configured by the user if bad record logger
is enabled or bad record action redirect. | |
 | carbon.enable.vector.reader | true | This parameter increases the performance of select
queries as it fetch columnar batch of size 4*1024 rows instead of fetching data row by row.
| |

http://git-wip-us.apache.org/repos/asf/carbondata/blob/7d3bf050/docs/dml-operation-on-carbondata.md
----------------------------------------------------------------------
diff --git a/docs/dml-operation-on-carbondata.md b/docs/dml-operation-on-carbondata.md
index 794bbc2..66109e8 100644
--- a/docs/dml-operation-on-carbondata.md
+++ b/docs/dml-operation-on-carbondata.md
@@ -198,7 +198,7 @@ OPTIONS('BAD_RECORDS_LOGGER_ENABLE'='true',
  | Options                   | Default Value | Description                              
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                       |
  |---------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
  | BAD_RECORDS_LOGGER_ENABLE | false         | Whether to create logs with details about
bad records.                                                                             
                                                                                         
                                                                                         
                                                                                         
                                       |
- | BAD_RECORDS_ACTION        | FAIL          | Following are the four types of action for
bad records:  FORCE: Auto-corrects the data by storing the bad records as NULL.  REDIRECT:
Bad records are written to the raw CSV instead of being loaded.  IGNORE: Bad records are neither
loaded nor written to the raw CSV.  FAIL: Data loading fails if any bad records are found.
 NOTE: In loaded data, if all records are bad records, the BAD_RECORDS_ACTION is invalid and
the load operation fails. |
+ | BAD_RECORDS_ACTION        | FORCE          | Following are the four types of action for
bad records:  FORCE: Auto-corrects the data by storing the bad records as NULL.  REDIRECT:
Bad records are written to the raw CSV instead of being loaded.  IGNORE: Bad records are neither
loaded nor written to the raw CSV.  FAIL: Data loading fails if any bad records are found.
 NOTE: In loaded data, if all records are bad records, the BAD_RECORDS_ACTION is invalid and
the load operation fails. |
  | IS_EMPTY_DATA_BAD_RECORD  | false         | If false, then empty ("" or '' or ,,) data
will not be considered as bad record and vice versa.                                     
                                                                                         
                                                                                         
                                                                                         
                                      |
  | BAD_RECORD_PATH           | -             | Specifies the HDFS path where bad records
are stored. By default the value is Null. This path must to be configured by the user if bad
record logger is enabled or bad record action redirect.                                  
                                                                                         
                                                                                         
                                    |
 

http://git-wip-us.apache.org/repos/asf/carbondata/blob/7d3bf050/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
----------------------------------------------------------------------
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
index 46a4f8c..4c6c56d 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestBatchSortDataLoad.scala
@@ -61,15 +61,11 @@ class TestBatchSortDataLoad extends QueryTest with BeforeAndAfterAll {
     sql("DROP TABLE IF EXISTS carbon_load6")
   }
 
-
-
   override def beforeAll {
     dropTable
     buildTestData
   }
-
-
-
+  
   test("test batch sort load by passing option to load command") {
 
     sql(


Mime
View raw message