kylin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shaofeng...@apache.org
Subject svn commit: r1696740 - in /incubator/kylin/site: ./ blog/2015/08/15/fast-cubing/ development/ docs/gettingstarted/ docs/howto/
Date Thu, 20 Aug 2015 09:37:16 GMT
Author: shaofengshi
Date: Thu Aug 20 09:37:16 2015
New Revision: 1696740

URL: http://svn.apache.org/r1696740
Log:
small update on fast cubing blog

Modified:
    incubator/kylin/site/blog/2015/08/15/fast-cubing/index.html
    incubator/kylin/site/development/dev_env.html
    incubator/kylin/site/development/howto_contribute.html
    incubator/kylin/site/development/howto_docs.html
    incubator/kylin/site/development/howto_package.html
    incubator/kylin/site/development/howto_release.html
    incubator/kylin/site/development/index.html
    incubator/kylin/site/development/new_metadata.html
    incubator/kylin/site/development/test_minicluster.html
    incubator/kylin/site/development/web_tech.html
    incubator/kylin/site/docs/gettingstarted/events.html
    incubator/kylin/site/docs/howto/howto_use_restapi_in_js.html
    incubator/kylin/site/feed.xml

Modified: incubator/kylin/site/blog/2015/08/15/fast-cubing/index.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/blog/2015/08/15/fast-cubing/index.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/blog/2015/08/15/fast-cubing/index.html (original)
+++ incubator/kylin/site/blog/2015/08/15/fast-cubing/index.html Thu Aug 20 09:37:16 2015
@@ -41,7 +41,7 @@
   <meta name="viewport" content="width=device-width, initial-scale=1">
 
   <title>Apache Kylin | Fast Cubing Algorithm in Apache Kylin: Concept</title>
-  <meta name="description" content="This article is the first post of a series document that introducing the fast cubing algorithm in Apache Kylin; The new algorithm will be released in the nea...">
+  <meta name="description" content="This article is the first post of a series document that introducing the fast cubing algorithm in Apache Kylin; The new algorithm will be released in the fut...">
   <meta name="author"      content="Apache Kylin">
   <link rel="shortcut icon" href="fav.png" type="image/png">
 
@@ -185,7 +185,7 @@
   </header>
 
   <article class="post-content" >
-    <p>This article is the first post of a series document that introducing the fast cubing algorithm in Apache Kylin; The new algorithm will be released in the near future (v1.1 or later); Now it is under internal testing and pilot; All the source code has been published in Kylin’s Git repository <a href="https://github.com/apache/incubator-kylin/">https://github.com/apache/incubator-kylin/</a> on 0.8 branch.</p>
+    <p>This article is the first post of a series document that introducing the fast cubing algorithm in Apache Kylin; The new algorithm will be released in the future release; Now it is under internal testing and pilot; All the source code has been published in Kylin’s Git repository <a href="https://github.com/apache/incubator-kylin/">https://github.com/apache/incubator-kylin/</a> on 0.8 branch.</p>
 
 <h2 id="by-layer-cubing-algorithm">By Layer Cubing Algorithm</h2>
 <p>Before introduce the fast cubing, I’d like to make a brief introduction on the as-is cubing algorithm, which is called “by layer cubing”;</p>
@@ -206,7 +206,7 @@ Figure 1: By layer cubing</p>
 <p><strong>Advantage</strong></p>
 
 <ul>
-  <li>This algorithm is clear as it mostly leverage the capability of Hadoop; The sorting, grouping and shuffling are all taken care by Hadoop, developer just focus on the cubing logic;</li>
+  <li>This algorithm is clear as it mostly leverage the capability of Hadoop MapReduce; The sorting, grouping and shuffling are all taken care by MapReduce, developer just focus on the cubing logic;</li>
   <li>Benefiting from Hadoop’s maturity, this algorithm is very stable; In our experience there is seldom case that the mapper or reducer could fail; Even if your Hadoop cluster is small or is busy, it can finish in the end;</li>
 </ul>
 
@@ -214,9 +214,9 @@ Figure 1: By layer cubing</p>
 
 <ul>
   <li>If the cube has many dimensions, it need the same number of MR jobs; As each Hadoop job scheduling need extra resource, the overhead cost to Hadoop is considerable;</li>
-  <li>This algorithm causes too much shuffling to Hadoop; The mapper doesn’t do aggregation, all the records that having same dimension values in next layer will be omitted to Hadoop, and then sent to reducer for aggregation;</li>
-  <li>Many writes to HDFS: each layer’s cubing need write its output to HDFS for next layer MR to consume; In the end, Kylin need another round MR to convert these output files to HBase HFile for bulk load; These jobs generates many intermediate files in HDFS;</li>
-  <li>All in all: the performance is not good, especially when the cube has many dimensions (&gt;10);</li>
+  <li>This algorithm causes too much shuffling to Hadoop; The mapper doesn’t do aggregation, all the records that having same dimension values in next layer will be omitted to Hadoop, and then aggregated by combiner and reducer;</li>
+  <li>Many reads/writes on HDFS: each layer’s cubing need write its output to HDFS for next layer MR to consume; In the end, Kylin need another round MR to convert these output files to HBase HFile for bulk load; These jobs generates many intermediate files in HDFS;</li>
+  <li>All in all: the performance is not good, especially when the cube has many dimensions;</li>
 </ul>
 
 <h2 id="fast-cubing-algorithm">Fast Cubing Algorithm</h2>
@@ -240,21 +240,21 @@ Figure 2: By segment cubing</p>
     <p>When aggregate the source records to base cuboid (1111), with the old “by layer” algorithm, the mapper will output 1 million records to Hadoop; With the fast cubing algorithm, after the pre-aggregation, it only outputs the number of [distinct A, B, C, D] records to Hadoop, which is certainly smaller than source data; In a normal case, it can be 1/10 to 1/1000 of the source records size;</p>
   </li>
   <li>
-    <p>When aggregate from parent to a child cuboid, say from base cuboid (1111) to 3-dimension cuboid 0111, the dimension A will be aggregated; We assume the dimension A is independent with other dimensions, after aggregation, the cuboid 0111’s size will be about 1/Card(A) of the base cuboid; So the output will be reduced to 1/Card(A) of the original one in this step;</p>
+    <p>When aggregate from parent to a child cuboid, say from base cuboid (1111) to 3-dimension cuboid 0111, the dimension A will be aggregated; We assume the dimension A is independent with other dimensions, after aggregation, the cuboid 0111’s size will be about 1/Card(A) of the base cuboid; So the output will be reduced to 1/Card(A) of the original one in this step.</p>
   </li>
 </ul>
 
-<p>Totally, assume the average cardinality of the dimensions is Card(N), the records that written from mapper to reducer can be reduced to 1/Card(N) of origin size; The less output to Hadoop, the less I/O and computing, the better performance;</p>
+<p>Totally, assume the average cardinality of the dimensions is Card(N), the records that written from mapper to reducer can be reduced to 1/Card(N) of origin size; The less output to Hadoop, the less I/O and computing, the better performance.</p>
 
 <h3 id="cuboid-spanning-tree">Cuboid spanning tree</h3>
-<p>Inner the mapper, there is another change in the cuboid spanning tree visiting order; In origin cubing, Kylin calculates the cuboids with Broad First Search order; In the fast cubing, it uses Depth First Search order, to reduce the cuboids that need be cached in memory;</p>
+<p>Inner the mapper, there is another change in the cuboid spanning tree visiting order; In origin cubing, Kylin calculates the cuboids with Broad First Search order; In the fast cubing, it uses Depth First Search order, to reduce the cuboids that need be cached in memory.</p>
 
-<p>Let’s look at the figure 3, which is a full spanning tree of a 4-dimension cube: before 0-Dimenson cuboid “*” be aggregated, cuboid “ABCD”, “BCD”, “CD” and “D” will be cached in memory; As “*” has no child, it will be outputted once be calculated, then comes “D”; After “C” be outputted, “CD” will be outputted as all of its children has finished; The memory for a cuboid can be released once it be outputted; The base cuboid ABCD will be the last one to finish;</p>
+<p>Let’s look at the figure 3, which is a full spanning tree of a 4-dimension cube: before 0-Dimenson cuboid “*” be aggregated, cuboid “ABCD”, “BCD”, “CD” and “D” will be cached in memory; As “*” has no child, it will be outputted once be calculated, then comes “D”; After “C” be outputted, “CD” will be outputted as all of its children has finished; The memory for a cuboid can be released once it be outputted; The base cuboid ABCD will be the last one to output.</p>
 
 <p><img src="/images/blog/cube-spanning-tree.png" alt="" /><br />
 Figure 3: Cuboid spanning tree</p>
 
-<p>With the DFS visiting order, the output from a mapper is almost fully sorted, as the cuboid ID is at the beginning position of row key, and inner a cuboid the rows are already sorted:</p>
+<p>With the DFS visiting order, the output from a mapper is fully sorted (except some special cases), as the cuboid ID is at the beginning position of row key, and inner a cuboid the rows are already sorted:</p>
 
 <div class="highlighter-rouge"><pre class="highlight"><code>0000
 0001[D0]
@@ -271,9 +271,9 @@ Figure 3: Cuboid spanning tree</p>
 ....
 </code></pre>
 </div>
-<p>Since the outputs from mapper are already sorted, Hadoop’s shuffle and sort would be faster;</p>
+<p>Since the outputs from mapper are already sorted, Hadoop’s sort would be more efficient;</p>
 
-<p>Besides, mapper’s pre-aggregation happens in memory, this avoids unnecessary disk and network I/O, and the overhead to Hadoop is considerable reduced;</p>
+<p>Besides, mapper’s pre-aggregation happens in memory, this avoids unnecessary disk and network I/O, and the overhead to Hadoop is reduced;</p>
 
 <p>###OutOfMemory error<br />
 During the development phase, we encountered the OutOfMemory error in mappers; this could happen when:</p>
@@ -281,6 +281,7 @@ During the development phase, we encount
 <div class="highlighter-rouge"><pre class="highlight"><code>a) The mapper's JVM heap size is small;
 b) "Distinct count" measure is used (HyperLogLog is space consuming) 
 c) The spanning tree is too deep (too many dimensions);
+d) The data blog feed to a mapper is too big;
 </code></pre>
 </div>
 
@@ -305,7 +306,7 @@ c) The spanning tree is too deep (too ma
 
 <h2 id="other-enhancements-in-fast-cubing">Other Enhancements in Fast Cubing</h2>
 
-<p>Together with the fast cubing algorithm, we also introduced several other enhancements in the cube build flow; I’s like to elaborate in next articles, please keep posted here; If you have interest on Apache Kylin, welcome to visit our home page <a href="http://kylin.incubator.apache.org/">http://kylin.incubator.apache.org/</a> and subscribe our development mailing list at <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
+<p>Together with the fast cubing algorithm, we also introduced several other enhancements in the cube build flow, like estimating the region splits by sampling, directly outputting HFile, and merging Cube over HBase Table, etc; I’d like to elaborate in the next articles, please keep posted here; If you have interest on Apache Kylin, welcome to visit our home page <a href="http://kylin.incubator.apache.org/">http://kylin.incubator.apache.org/</a> and subscribe our development mailing list at <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#10
 4;&#101;&#046;&#111;&#114;&#103;</a></p>
 
   </article>
 

Modified: incubator/kylin/site/development/dev_env.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/dev_env.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/dev_env.html (original)
+++ incubator/kylin/site/development/dev_env.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/howto_contribute.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/howto_contribute.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/howto_contribute.html (original)
+++ incubator/kylin/site/development/howto_contribute.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/howto_docs.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/howto_docs.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/howto_docs.html (original)
+++ incubator/kylin/site/development/howto_docs.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/howto_package.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/howto_package.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/howto_package.html (original)
+++ incubator/kylin/site/development/howto_package.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/howto_release.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/howto_release.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/howto_release.html (original)
+++ incubator/kylin/site/development/howto_release.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/index.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/index.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/index.html (original)
+++ incubator/kylin/site/development/index.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/new_metadata.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/new_metadata.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/new_metadata.html (original)
+++ incubator/kylin/site/development/new_metadata.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/test_minicluster.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/test_minicluster.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/test_minicluster.html (original)
+++ incubator/kylin/site/development/test_minicluster.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/development/web_tech.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/development/web_tech.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/development/web_tech.html (original)
+++ incubator/kylin/site/development/web_tech.html Thu Aug 20 09:37:16 2015
@@ -205,6 +205,8 @@
     
   
     
+  
+    
       <li><a href="/development/index.html" class="list-group-item-lay pjaxlink">Development Quick Guide</a></li>      
       
 
@@ -220,6 +222,8 @@
 
   
     
+  
+    
       <li><a href="/development/dev_env.html" class="list-group-item-lay pjaxlink">Setup Development Env</a></li>      
       
 
@@ -237,6 +241,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_contribute.html" class="list-group-item-lay pjaxlink">How to Contribute</a></li>      
       
 
@@ -256,6 +262,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_docs.html" class="list-group-item-lay pjaxlink">How to Write Document</a></li>      
       
 
@@ -277,6 +285,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_package.html" class="list-group-item-lay pjaxlink">How to Build Binary Package</a></li>      
       
 
@@ -300,6 +310,8 @@
     
   
     
+  
+    
       <li><a href="/development/howto_release.html" class="list-group-item-lay pjaxlink">How to Making a Release</a></li>      
       
 
@@ -327,6 +339,8 @@
     
   
     
+  
+    
       <li><a href="/development/new_metadata.html" class="list-group-item-lay pjaxlink">New Metadata Model</a></li>      
       
 
@@ -358,6 +372,8 @@
     
   
     
+  
+    
       <li><a href="/development/web_tech.html" class="list-group-item-lay pjaxlink">Kylin Web Summary</a></li>      
       
 
@@ -387,10 +403,27 @@
     
   
     
+  
+    
       <li><a href="/development/test_minicluster.html" class="list-group-item-lay pjaxlink">Run Kylin Test Case with Mini Cluster</a></li>      
       
 
 
+   
+  
+
+  
+    
+  
+
+
+
+  
+    
+      <li><a href="/development/about_temp_files.html" class="list-group-item-lay pjaxlink">About Temp Files</a></li>      
+      
+
+
 
 
 

Modified: incubator/kylin/site/docs/gettingstarted/events.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/docs/gettingstarted/events.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/docs/gettingstarted/events.html (original)
+++ incubator/kylin/site/docs/gettingstarted/events.html Thu Aug 20 09:37:16 2015
@@ -1384,6 +1384,7 @@
 <p><strong>Conferences</strong></p>
 
 <ul>
+  <li><a href="http://www.chinahadoop.com/2015/July/Shanghai/agenda.php">Apache Kylin - Balance Between Space and Time</a> (<a href="http://www.slideshare.net/qhzhou/apache-kylin-china-hadoop-summit-2015-shanghai">slides</a>) by <a href="https://github.com/qhzhou">Qianhao Zhou</a>, at Hadoop Summit 2015 in Shanghai, China, 2015-07-24</li>
   <li><a href="https://www.youtube.com/watch?v=jgvZSFaXPgI">Apache Kylin - Balance Between Space and Time</a>, <a href="http://www.slideshare.net/DebashisSaha/apache-kylin-balance-between-space-and-time-hadop-summit-2015">slides</a> (<a href="https://www.youtube.com/watch?v=jgvZSFaXPgI">video</a>, <a href="http://www.slideshare.net/DebashisSaha/apache-kylin-balance-between-space-and-time-hadop-summit-2015">slides</a>) by <a href="https://twitter.com/debashis_saha">Debashis Saha</a> &amp; <a href="https://twitter.com/lukehq">Luke Han</a>, at Hadoop Summit 2015 in San Jose, US, 2015-06-09</li>
   <li><a href="https://vimeo.com/128152444">HBaseCon 2015: Apache Kylin; Extreme OLAP Engine for Hadoop</a> (<a href="https://vimeo.com/128152444">video</a>, <a href="http://www.slideshare.net/HBaseCon/ecosystem-session-3b">slides</a>) by <a href="https://twitter.com/SeshuAd">Seshu Adunuthula</a> at HBaseCon 2015 in San Francisco, US, 2015-05-07</li>
   <li><a href="http://strataconf.com/big-data-conference-uk-2015/public/schedule/detail/40029">Apache Kylin - Extreme OLAP Engine for Hadoop</a> (<a href="http://www.slideshare.net/lukehan/apache-kylin-extreme-olap-engine-for-big-data">slides</a>) by <a href="https://twitter.com/lukehq">Luke Han</a> &amp; <a href="https://github.com/liyang-gmt8">Yang Li</a>, at Strata+Hadoop World in London, UK, 2015-05-06</li>

Modified: incubator/kylin/site/docs/howto/howto_use_restapi_in_js.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/docs/howto/howto_use_restapi_in_js.html?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/docs/howto/howto_use_restapi_in_js.html (original)
+++ incubator/kylin/site/docs/howto/howto_use_restapi_in_js.html Thu Aug 20 09:37:16 2015
@@ -1385,7 +1385,7 @@
     var request = $.ajax({
        url: "http://hostname/kylin/api/query",
        type: "POST",
-       data: '{"sql":"select count(*) from SUMMARY;","offset":0,"limit":50000,"partial":true,"project":"test"}',
+       data: '{"sql":"select count(*) from SUMMARY;","offset":0,"limit":50000,"acceptPartial":true,"project":"test"}',
        dataType: "json"
     });
     request.done(function( msg ) {

Modified: incubator/kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/incubator/kylin/site/feed.xml?rev=1696740&r1=1696739&r2=1696740&view=diff
==============================================================================
--- incubator/kylin/site/feed.xml (original)
+++ incubator/kylin/site/feed.xml Thu Aug 20 09:37:16 2015
@@ -19,13 +19,13 @@
     <description>Apache Kylin Home</description>
     <link>http://kylin.incubator.apache.org/</link>
     <atom:link href="http://kylin.incubator.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
-    <pubDate>Mon, 17 Aug 2015 00:19:38 -0700</pubDate>
-    <lastBuildDate>Mon, 17 Aug 2015 00:19:38 -0700</lastBuildDate>
+    <pubDate>Thu, 20 Aug 2015 02:35:15 -0700</pubDate>
+    <lastBuildDate>Thu, 20 Aug 2015 02:35:15 -0700</lastBuildDate>
     <generator>Jekyll v2.5.3</generator>
     
       <item>
         <title>Fast Cubing Algorithm in Apache Kylin: Concept</title>
-        <description>&lt;p&gt;This article is the first post of a series document that introducing the fast cubing algorithm in Apache Kylin; The new algorithm will be released in the near future (v1.1 or later); Now it is under internal testing and pilot; All the source code has been published in Kylin’s Git repository &lt;a href=&quot;https://github.com/apache/incubator-kylin/&quot;&gt;https://github.com/apache/incubator-kylin/&lt;/a&gt; on 0.8 branch.&lt;/p&gt;
+        <description>&lt;p&gt;This article is the first post of a series document that introducing the fast cubing algorithm in Apache Kylin; The new algorithm will be released in the future release; Now it is under internal testing and pilot; All the source code has been published in Kylin’s Git repository &lt;a href=&quot;https://github.com/apache/incubator-kylin/&quot;&gt;https://github.com/apache/incubator-kylin/&lt;/a&gt; on 0.8 branch.&lt;/p&gt;
 
 &lt;h2 id=&quot;by-layer-cubing-algorithm&quot;&gt;By Layer Cubing Algorithm&lt;/h2&gt;
 &lt;p&gt;Before introduce the fast cubing, I’d like to make a brief introduction on the as-is cubing algorithm, which is called “by layer cubing”;&lt;/p&gt;
@@ -46,7 +46,7 @@ Figure 1: By layer cubing&lt;/p&gt;
 &lt;p&gt;&lt;strong&gt;Advantage&lt;/strong&gt;&lt;/p&gt;
 
 &lt;ul&gt;
-  &lt;li&gt;This algorithm is clear as it mostly leverage the capability of Hadoop; The sorting, grouping and shuffling are all taken care by Hadoop, developer just focus on the cubing logic;&lt;/li&gt;
+  &lt;li&gt;This algorithm is clear as it mostly leverage the capability of Hadoop MapReduce; The sorting, grouping and shuffling are all taken care by MapReduce, developer just focus on the cubing logic;&lt;/li&gt;
   &lt;li&gt;Benefiting from Hadoop’s maturity, this algorithm is very stable; In our experience there is seldom case that the mapper or reducer could fail; Even if your Hadoop cluster is small or is busy, it can finish in the end;&lt;/li&gt;
 &lt;/ul&gt;
 
@@ -54,9 +54,9 @@ Figure 1: By layer cubing&lt;/p&gt;
 
 &lt;ul&gt;
   &lt;li&gt;If the cube has many dimensions, it need the same number of MR jobs; As each Hadoop job scheduling need extra resource, the overhead cost to Hadoop is considerable;&lt;/li&gt;
-  &lt;li&gt;This algorithm causes too much shuffling to Hadoop; The mapper doesn’t do aggregation, all the records that having same dimension values in next layer will be omitted to Hadoop, and then sent to reducer for aggregation;&lt;/li&gt;
-  &lt;li&gt;Many writes to HDFS: each layer’s cubing need write its output to HDFS for next layer MR to consume; In the end, Kylin need another round MR to convert these output files to HBase HFile for bulk load; These jobs generates many intermediate files in HDFS;&lt;/li&gt;
-  &lt;li&gt;All in all: the performance is not good, especially when the cube has many dimensions (&amp;gt;10);&lt;/li&gt;
+  &lt;li&gt;This algorithm causes too much shuffling to Hadoop; The mapper doesn’t do aggregation, all the records that having same dimension values in next layer will be omitted to Hadoop, and then aggregated by combiner and reducer;&lt;/li&gt;
+  &lt;li&gt;Many reads/writes on HDFS: each layer’s cubing need write its output to HDFS for next layer MR to consume; In the end, Kylin need another round MR to convert these output files to HBase HFile for bulk load; These jobs generates many intermediate files in HDFS;&lt;/li&gt;
+  &lt;li&gt;All in all: the performance is not good, especially when the cube has many dimensions;&lt;/li&gt;
 &lt;/ul&gt;
 
 &lt;h2 id=&quot;fast-cubing-algorithm&quot;&gt;Fast Cubing Algorithm&lt;/h2&gt;
@@ -80,21 +80,21 @@ Figure 2: By segment cubing&lt;/p&gt;
     &lt;p&gt;When aggregate the source records to base cuboid (1111), with the old “by layer” algorithm, the mapper will output 1 million records to Hadoop; With the fast cubing algorithm, after the pre-aggregation, it only outputs the number of [distinct A, B, C, D] records to Hadoop, which is certainly smaller than source data; In a normal case, it can be 1/10 to 1/1000 of the source records size;&lt;/p&gt;
   &lt;/li&gt;
   &lt;li&gt;
-    &lt;p&gt;When aggregate from parent to a child cuboid, say from base cuboid (1111) to 3-dimension cuboid 0111, the dimension A will be aggregated; We assume the dimension A is independent with other dimensions, after aggregation, the cuboid 0111’s size will be about 1/Card(A) of the base cuboid; So the output will be reduced to 1/Card(A) of the original one in this step;&lt;/p&gt;
+    &lt;p&gt;When aggregate from parent to a child cuboid, say from base cuboid (1111) to 3-dimension cuboid 0111, the dimension A will be aggregated; We assume the dimension A is independent with other dimensions, after aggregation, the cuboid 0111’s size will be about 1/Card(A) of the base cuboid; So the output will be reduced to 1/Card(A) of the original one in this step.&lt;/p&gt;
   &lt;/li&gt;
 &lt;/ul&gt;
 
-&lt;p&gt;Totally, assume the average cardinality of the dimensions is Card(N), the records that written from mapper to reducer can be reduced to 1/Card(N) of origin size; The less output to Hadoop, the less I/O and computing, the better performance;&lt;/p&gt;
+&lt;p&gt;Totally, assume the average cardinality of the dimensions is Card(N), the records that written from mapper to reducer can be reduced to 1/Card(N) of origin size; The less output to Hadoop, the less I/O and computing, the better performance.&lt;/p&gt;
 
 &lt;h3 id=&quot;cuboid-spanning-tree&quot;&gt;Cuboid spanning tree&lt;/h3&gt;
-&lt;p&gt;Inner the mapper, there is another change in the cuboid spanning tree visiting order; In origin cubing, Kylin calculates the cuboids with Broad First Search order; In the fast cubing, it uses Depth First Search order, to reduce the cuboids that need be cached in memory;&lt;/p&gt;
+&lt;p&gt;Inner the mapper, there is another change in the cuboid spanning tree visiting order; In origin cubing, Kylin calculates the cuboids with Broad First Search order; In the fast cubing, it uses Depth First Search order, to reduce the cuboids that need be cached in memory.&lt;/p&gt;
 
-&lt;p&gt;Let’s look at the figure 3, which is a full spanning tree of a 4-dimension cube: before 0-Dimenson cuboid “*” be aggregated, cuboid “ABCD”, “BCD”, “CD” and “D” will be cached in memory; As “*” has no child, it will be outputted once be calculated, then comes “D”; After “C” be outputted, “CD” will be outputted as all of its children has finished; The memory for a cuboid can be released once it be outputted; The base cuboid ABCD will be the last one to finish;&lt;/p&gt;
+&lt;p&gt;Let’s look at the figure 3, which is a full spanning tree of a 4-dimension cube: before 0-Dimenson cuboid “*” be aggregated, cuboid “ABCD”, “BCD”, “CD” and “D” will be cached in memory; As “*” has no child, it will be outputted once be calculated, then comes “D”; After “C” be outputted, “CD” will be outputted as all of its children has finished; The memory for a cuboid can be released once it be outputted; The base cuboid ABCD will be the last one to output.&lt;/p&gt;
 
 &lt;p&gt;&lt;img src=&quot;/images/blog/cube-spanning-tree.png&quot; alt=&quot;&quot; /&gt;&lt;br /&gt;
 Figure 3: Cuboid spanning tree&lt;/p&gt;
 
-&lt;p&gt;With the DFS visiting order, the output from a mapper is almost fully sorted, as the cuboid ID is at the beginning position of row key, and inner a cuboid the rows are already sorted:&lt;/p&gt;
+&lt;p&gt;With the DFS visiting order, the output from a mapper is fully sorted (except some special cases), as the cuboid ID is at the beginning position of row key, and inner a cuboid the rows are already sorted:&lt;/p&gt;
 
 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;0000
 0001[D0]
@@ -111,9 +111,9 @@ Figure 3: Cuboid spanning tree&lt;/p&gt;
 ....
 &lt;/code&gt;&lt;/pre&gt;
 &lt;/div&gt;
-&lt;p&gt;Since the outputs from mapper are already sorted, Hadoop’s shuffle and sort would be faster;&lt;/p&gt;
+&lt;p&gt;Since the outputs from mapper are already sorted, Hadoop’s sort would be more efficient;&lt;/p&gt;
 
-&lt;p&gt;Besides, mapper’s pre-aggregation happens in memory, this avoids unnecessary disk and network I/O, and the overhead to Hadoop is considerable reduced;&lt;/p&gt;
+&lt;p&gt;Besides, mapper’s pre-aggregation happens in memory, this avoids unnecessary disk and network I/O, and the overhead to Hadoop is reduced;&lt;/p&gt;
 
 &lt;p&gt;###OutOfMemory error&lt;br /&gt;
 During the development phase, we encountered the OutOfMemory error in mappers; this could happen when:&lt;/p&gt;
@@ -121,6 +121,7 @@ During the development phase, we encount
 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;a) The mapper&#39;s JVM heap size is small;
 b) &quot;Distinct count&quot; measure is used (HyperLogLog is space consuming) 
 c) The spanning tree is too deep (too many dimensions);
+d) The data blog feed to a mapper is too big;
 &lt;/code&gt;&lt;/pre&gt;
 &lt;/div&gt;
 
@@ -145,7 +146,7 @@ c) The spanning tree is too deep (too ma
 
 &lt;h2 id=&quot;other-enhancements-in-fast-cubing&quot;&gt;Other Enhancements in Fast Cubing&lt;/h2&gt;
 
-&lt;p&gt;Together with the fast cubing algorithm, we also introduced several other enhancements in the cube build flow; I’s like to elaborate in next articles, please keep posted here; If you have interest on Apache Kylin, welcome to visit our home page &lt;a href=&quot;http://kylin.incubator.apache.org/&quot;&gt;http://kylin.incubator.apache.org/&lt;/a&gt; and subscribe our development mailing list at &lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#100;&amp;#101;&amp;#118;&amp;#045;&amp;#115;&amp;#117;&amp;#098;&amp;#115;&amp;#099;&amp;#114;&amp;#105;&amp;#098;&amp;#101;&amp;#064;&amp;#107;&amp;#121;&amp;#108;&amp;#105;&amp;#110;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&quot;&gt;&amp;#100;&amp;#101;&amp;#118;&amp;#045;&amp;#115;&amp;#117;&amp;#098;&amp;#115;&amp;#099;&amp;#11
 4;&amp;#105;&amp;#098;&amp;#101;&amp;#064;&amp;#107;&amp;#121;&amp;#108;&amp;#105;&amp;#110;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&lt;/a&gt;&lt;/p&gt;
+&lt;p&gt;Together with the fast cubing algorithm, we also introduced several other enhancements in the cube build flow, like estimating the region splits by sampling, directly outputting HFile, and merging Cube over HBase Table, etc; I’d like to elaborate in the next articles, please keep posted here; If you have interest on Apache Kylin, welcome to visit our home page &lt;a href=&quot;http://kylin.incubator.apache.org/&quot;&gt;http://kylin.incubator.apache.org/&lt;/a&gt; and subscribe our development mailing list at &lt;a href=&quot;&amp;#109;&amp;#097;&amp;#105;&amp;#108;&amp;#116;&amp;#111;:&amp;#100;&amp;#101;&amp;#118;&amp;#045;&amp;#115;&amp;#117;&amp;#098;&amp;#115;&amp;#099;&amp;#114;&amp;#105;&amp;#098;&amp;#101;&amp;#064;&amp;#107;&amp;#121;&amp;#108;&amp;#105;&amp;#110;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;
 &amp;#103;&quot;&gt;&amp;#100;&amp;#101;&amp;#118;&amp;#045;&amp;#115;&amp;#117;&amp;#098;&amp;#115;&amp;#099;&amp;#114;&amp;#105;&amp;#098;&amp;#101;&amp;#064;&amp;#107;&amp;#121;&amp;#108;&amp;#105;&amp;#110;&amp;#046;&amp;#105;&amp;#110;&amp;#099;&amp;#117;&amp;#098;&amp;#097;&amp;#116;&amp;#111;&amp;#114;&amp;#046;&amp;#097;&amp;#112;&amp;#097;&amp;#099;&amp;#104;&amp;#101;&amp;#046;&amp;#111;&amp;#114;&amp;#103;&lt;/a&gt;&lt;/p&gt;
 </description>
         <pubDate>Sat, 15 Aug 2015 05:28:00 -0700</pubDate>
         <link>http://kylin.incubator.apache.org/blog/2015/08/15/fast-cubing/</link>



Mime
View raw message