carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chenliang...@apache.org
Subject [1/2] incubator-carbondata-site git commit: update md files
Date Thu, 27 Apr 2017 07:01:07 GMT
Repository: incubator-carbondata-site
Updated Branches:
  refs/heads/asf-site 24e325ab0 -> b22227019


http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/b2222701/src/main/webapp/quick-start-guide.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/quick-start-guide.html b/src/main/webapp/quick-start-guide.html
index 0246f64..633bc6c 100644
--- a/src/main/webapp/quick-start-guide.html
+++ b/src/main/webapp/quick-start-guide.html
@@ -67,7 +67,7 @@
                                    target="_blank">Release Archive</a></li>
                         </ul>
                     </li>
-                    <li><a href="mainpage.html" class="">Documentation</a></li>
+                    <li><a href="mainpage.html" class="active">Documentation</a></li>
                     <li class="dropdown">
                         <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button"
aria-haspopup="true"
                            aria-expanded="false">Community <span class="caret"></span></a>
@@ -125,7 +125,7 @@
                             <tr>
                                 <td style="width:80%">
                                     <input type="text" name="q" size=" 5" maxlength="255"
value=""
-                                           class="search-input"/>
+                                           class="search-input"  placeholder="Search...."
   required/>
                                 </td>
                                 <td style="width:20%">
                                     <input type="submit" value="Search"/></td>
@@ -133,7 +133,7 @@
                             <tr>
                                 <td align="left" style="font-size:75%" colspan="2">
                                     <input type="checkbox" name="sitesearch" value="carbondata.apache.org"
checked/>
-                                    Only search for CarbonData
+                                    <span style=" position: relative; top: -3px;">
Only search for CarbonData</span>
                                 </td>
                             </tr>
                         </table>

http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/b2222701/src/main/webapp/supported-data-types-in-carbondata.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/supported-data-types-in-carbondata.html b/src/main/webapp/supported-data-types-in-carbondata.html
index b56bc59..9757fb3 100644
--- a/src/main/webapp/supported-data-types-in-carbondata.html
+++ b/src/main/webapp/supported-data-types-in-carbondata.html
@@ -67,7 +67,7 @@
                                    target="_blank">Release Archive</a></li>
                         </ul>
                     </li>
-                    <li><a href="mainpage.html" class="">Documentation</a></li>
+                    <li><a href="mainpage.html" class="active">Documentation</a></li>
                     <li class="dropdown">
                         <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button"
aria-haspopup="true"
                            aria-expanded="false">Community <span class="caret"></span></a>
@@ -125,7 +125,7 @@
                             <tr>
                                 <td style="width:80%">
                                     <input type="text" name="q" size=" 5" maxlength="255"
value=""
-                                           class="search-input"/>
+                                           class="search-input"  placeholder="Search...."
   required/>
                                 </td>
                                 <td style="width:20%">
                                     <input type="submit" value="Search"/></td>
@@ -133,7 +133,7 @@
                             <tr>
                                 <td align="left" style="font-size:75%" colspan="2">
                                     <input type="checkbox" name="sitesearch" value="carbondata.apache.org"
checked/>
-                                    Only search for CarbonData
+                                    <span style=" position: relative; top: -3px;">
Only search for CarbonData</span>
                                 </td>
                             </tr>
                         </table>

http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/b2222701/src/main/webapp/troubleshooting.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/troubleshooting.html b/src/main/webapp/troubleshooting.html
index 48f8871..7ae7dc7 100644
--- a/src/main/webapp/troubleshooting.html
+++ b/src/main/webapp/troubleshooting.html
@@ -67,7 +67,7 @@
                                    target="_blank">Release Archive</a></li>
                         </ul>
                     </li>
-                    <li><a href="mainpage.html" class="">Documentation</a></li>
+                    <li><a href="mainpage.html" class="active">Documentation</a></li>
                     <li class="dropdown">
                         <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button"
aria-haspopup="true"
                            aria-expanded="false">Community <span class="caret"></span></a>
@@ -125,7 +125,7 @@
                             <tr>
                                 <td style="width:80%">
                                     <input type="text" name="q" size=" 5" maxlength="255"
value=""
-                                           class="search-input"/>
+                                           class="search-input"  placeholder="Search...."
   required/>
                                 </td>
                                 <td style="width:20%">
                                     <input type="submit" value="Search"/></td>
@@ -133,7 +133,7 @@
                             <tr>
                                 <td align="left" style="font-size:75%" colspan="2">
                                     <input type="checkbox" name="sitesearch" value="carbondata.apache.org"
checked/>
-                                    Only search for CarbonData
+                                    <span style=" position: relative; top: -3px;">
Only search for CarbonData</span>
                                 </td>
                             </tr>
                         </table>

http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/b2222701/src/main/webapp/useful-tips-on-carbondata.html
----------------------------------------------------------------------
diff --git a/src/main/webapp/useful-tips-on-carbondata.html b/src/main/webapp/useful-tips-on-carbondata.html
index 39e6b3c..b5ff71c 100644
--- a/src/main/webapp/useful-tips-on-carbondata.html
+++ b/src/main/webapp/useful-tips-on-carbondata.html
@@ -67,7 +67,7 @@
                                    target="_blank">Release Archive</a></li>
                         </ul>
                     </li>
-                    <li><a href="mainpage.html" class="">Documentation</a></li>
+                    <li><a href="mainpage.html" class="active">Documentation</a></li>
                     <li class="dropdown">
                         <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button"
aria-haspopup="true"
                            aria-expanded="false">Community <span class="caret"></span></a>
@@ -125,7 +125,7 @@
                             <tr>
                                 <td style="width:80%">
                                     <input type="text" name="q" size=" 5" maxlength="255"
value=""
-                                           class="search-input"/>
+                                           class="search-input"  placeholder="Search...."
   required/>
                                 </td>
                                 <td style="width:20%">
                                     <input type="submit" value="Search"/></td>
@@ -133,7 +133,7 @@
                             <tr>
                                 <td align="left" style="font-size:75%" colspan="2">
                                     <input type="checkbox" name="sitesearch" value="carbondata.apache.org"
checked/>
-                                    Only search for CarbonData
+                                    <span style=" position: relative; top: -3px;">
Only search for CarbonData</span>
                                 </td>
                             </tr>
                         </table>
@@ -162,7 +162,8 @@
 The following sections will elaborate on the above topics :</p>
 <ul>
 <li><a href="#suggestions-to-create-carbondata-table">Suggestions to create CarbonData
Table</a></li>
-<li><a href="#configurations-for-optimizing-carbondata-performance">Configurations
For Optimizing CarbonData Performance</a></li>
+<li><a href="#configuration-for-optimizing-data-loading-performance-for-massive-data">Configuration
for Optimizing Data Loading performance for Massive Data</a></li>
+<li><a href="#optimizing-mass-data-loading">Optimizing Mass Data Loading</a></li>
 </ul>
 <h2>
 <a id="suggestions-to-create-carbondata-table" class="anchor" href="#suggestions-to-create-carbondata-table"
aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Suggestions
to Create CarbonData Table</h2>
@@ -236,7 +237,7 @@ The create table command can be modified as suggested below :</p>
 <pre><code>  create table carbondata_table(
   msisdn String,
   ...
-  )STORED BY 'org.apache.carbondata.format' 
+  )STORED BY 'org.apache.carbondata.format'
   TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,..',
   'DICTIONARY_INCLUDE'='...');
 </code></pre>
@@ -257,7 +258,7 @@ The create table command can be modified as suggested below :</p>
   HOST String,
   MSISDN String,
   ...
-  )STORED BY 'org.apache.carbondata.format' 
+  )STORED BY 'org.apache.carbondata.format'
   TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST..',
   'DICTIONARY_INCLUDE'='Dime_1..');
 </code></pre>
@@ -274,7 +275,7 @@ The create table command can be modified as below :</p>
   HOST String,
   MSISDN String,
   ...
-  )STORED BY 'org.apache.carbondata.format' 
+  )STORED BY 'org.apache.carbondata.format'
   TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI..',
   'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME..');
 </code></pre>
@@ -294,7 +295,7 @@ query performance. The create table command can be modified as below :</p>
   counter_2 double,
   ...
   counter_100 double
-  )STORED BY 'org.apache.carbondata.format' 
+  )STORED BY 'org.apache.carbondata.format'
   TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI',
   'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME');
 </code></pre>
@@ -316,9 +317,9 @@ suggested to put start_time at the end of dimensions.</p>
   BEGIN_TIME bigint,
   ...
   counter_100 double
-  )STORED BY 'org.apache.carbondata.format' 
+  )STORED BY 'org.apache.carbondata.format'
   TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI',
-  'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME'); 
+  'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME');
 </code></pre>
 <ul>
 <li>
@@ -331,6 +332,61 @@ excessive memory usage.</p>
 </li>
 </ul>
 <h2>
+<a id="configuration-for-optimizing-data-loading-performance-for-massive-data" class="anchor"
href="#configuration-for-optimizing-data-loading-performance-for-massive-data" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Configuration for
Optimizing Data Loading performance for Massive Data</h2>
+<p>CarbonData supports large data load, in this process sorting data while loading
consumes a lot of memory and disk IO and
+this can result sometimes in "Out Of Memory" exception.
+If you do not have much memory to use, then you may prefer to slow the speed of data loading
instead of data load failure.
+You can configure CarbonData by tuning following properties in carbon.properties file to
get a better performance.:</p>
+<table>
+<thead>
+<tr>
+<th>Parameter</th>
+<th>Default Value</th>
+<th>Description/Tuning</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>carbon.number.of.cores.while.loading</td>
+<td>Default: 2.This value should be &gt;= 2</td>
+<td>Specifies the number of cores used for data processing during data loading in CarbonData.</td>
+</tr>
+<tr>
+<td>carbon.sort.size</td>
+<td>Data loading</td>
+<td>Default: 100000. The value should be &gt;= 100.</td>
+</tr>
+<tr>
+<td>carbon.sort.file.write.buffer.size</td>
+<td>Default:  50000.</td>
+<td>DataOutputStream buffer.</td>
+</tr>
+<tr>
+<td>carbon.number.of.cores.block.sort</td>
+<td>Default: 7</td>
+<td>If you have huge memory and cpus, increase it as you will</td>
+</tr>
+<tr>
+<td>carbon.merge.sort.reader.thread</td>
+<td>Default: 3</td>
+<td>Specifies the number of cores used for temp file merging during data loading in
CarbonData.</td>
+</tr>
+<tr>
+<td>carbon.merge.sort.prefetch</td>
+<td>Default: true</td>
+<td>You may want set this value to false if you have not enough memory</td>
+</tr>
+</tbody>
+</table>
+<p>For example, if there are  10 million records ,and i have only 16 cores ,64GB memory,
will be loaded to CarbonData table.
+Using the default configuration  always fail in sort step. Modify carbon.properties as suggested
below</p>
+<pre><code>carbon.number.of.cores.block.sort=1
+carbon.merge.sort.reader.thread=1
+carbon.sort.size=5000
+carbon.sort.file.write.buffer.size=5000
+carbon.merge.sort.prefetch=false
+</code></pre>
+<h2>
 <a id="configurations-for-optimizing-carbondata-performance" class="anchor" href="#configurations-for-optimizing-carbondata-performance"
aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configurations
for Optimizing CarbonData Performance</h2>
 <p>Recently we did some performance POC on CarbonData for Finance and telecommunication
Field. It involved detailed queries and aggregation
 scenarios. After the completion of POC, some of the configurations impacting the performance
have been identified and tabulated below :</p>
@@ -396,6 +452,8 @@ scenarios. After the completion of POC, some of the configurations impacting
the
 </tr>
 </tbody>
 </table>
+<p>Note: If your CarbonData instance is provided only for query, you may specify the
conf 'spark.speculation=true' which is conf
+in spark.</p>
 </div>
 </div>
 </div>


Mime
View raw message