kylin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mahong...@apache.org
Subject svn commit: r1692810 - in /incubator/kylin/site: docs/development/dev_env.html docs/development/metadata_store.html docs/howto/howto_docs.html docs/install/kylin_docker.html feed.xml
Date Mon, 27 Jul 2015 05:37:41 GMT
Author: mahongbin
Date: Mon Jul 27 05:37:41 2015
New Revision: 1692810

URL: http://svn.apache.org/r1692810
Log:
KYLIN-908

Modified:
    incubator/kylin/site/docs/development/dev_env.html
    incubator/kylin/site/docs/development/metadata_store.html
    incubator/kylin/site/docs/howto/howto_docs.html
    incubator/kylin/site/docs/install/kylin_docker.html
    incubator/kylin/site/feed.xml

Modified: incubator/kylin/site/docs/development/dev_env.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/docs/development/dev_env.html?rev=1692810&r1=1692809&r2=1692810&view=diff
==============================================================================
--- incubator/kylin/site/docs/development/dev_env.html (original)
+++ incubator/kylin/site/docs/development/dev_env.html Mon Jul 27 05:37:41 2015
@@ -2152,14 +2152,14 @@
 
 <h2 id="environment-on-the-hadoop-cli">Environment on the Hadoop CLI</h2>
 
-<p>Off-Hadoop-CLI installation requires you having a hadoop CLI machine (or a hadoop
sandbox) as well as your local develop machine. To make things easier we strongly recommend
you starting with running Kylin on a hadoop sandbox, like <a href="http://hortonworks.com/products/hortonworks-sandbox/">http://hortonworks.com/products/hortonworks-sandbox/</a>.
In the following tutorial we’ll go with <strong>Hortonworks Sandbox 2.2</strong>.</p>
+<p>Off-Hadoop-CLI installation requires you having a hadoop CLI machine (or a hadoop
sandbox) as well as your local develop machine. To make things easier we strongly recommend
you starting with running Kylin on a hadoop sandbox, like <a href="http://hortonworks.com/products/hortonworks-sandbox/">http://hortonworks.com/products/hortonworks-sandbox/</a>.
In the following tutorial we’ll go with <strong>Hortonworks Sandbox 2.2.4</strong>.
It is recommended that you provide enough memory to your sandbox, 8G or more is preferred.</p>
 
 <h3 id="start-hadoop">Start Hadoop</h3>
 
 <p>In Hortonworks sandbox, ambari helps to launch hadoop:</p>
 
 <div class="highlight"><pre><code class="language-groff" data-lang="groff">ambari-agent
start
-	ambari-server start</code></pre></div>
+ambari-server start</code></pre></div>
 
 <p>With both command successfully run you can go to ambari home page at <a href="http://yoursandboxip:8080">http://yoursandboxip:8080</a>
(user:admin,password:admin) to check everything’s status. By default ambari disables
Hbase, you’ll need manually start the <code class="highlighter-rouge">Hbase</code>
service.</p>
 
@@ -2172,15 +2172,15 @@
 <p>The latest maven can be found at <a href="http://maven.apache.org/download.cgi">http://maven.apache.org/download.cgi</a>,
we create a symbolic so that <code class="highlighter-rouge">mvn</code> can be
run anywhere.</p>
 
 <div class="highlight"><pre><code class="language-groff" data-lang="groff">cd
~
-	wget http://xenia.sote.hu/ftp/mirrors/www.apache.org/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
-	tar -xzvf apache-maven-3.2.5-bin.tar.gz
-	ln -s /root/apache-maven-3.2.5/bin/mvn /usr/bin/mvn</code></pre></div>
+wget http://xenia.sote.hu/ftp/mirrors/www.apache.org/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
+tar -xzvf apache-maven-3.2.5-bin.tar.gz
+ln -s /root/apache-maven-3.2.5/bin/mvn /usr/bin/mvn</code></pre></div>
 
 <h3 id="compile">Compile</h3>
 
 <p>First clone the Kylin project to your local:</p>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">git
clone https://github.com/KylinOLAP/Kylin.git</code></pre></div>
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">git
clone https://github.com/apache/incubator-kylin.git</code></pre></div>
 
 <p>Install Kylin artifacts to the maven repo</p>
 
@@ -2209,20 +2209,27 @@
 
 <h3 id="run-unit-tests">Run unit tests</h3>
 
-<p>Run a end-to-end cube building test</p>
+<p>Run a end-to-end cube building test, these special test cases will populate some
sample cubes in your metadata store and build them ready.<br />
+It might take a while (maybe one hour), please keep patient.</p>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">mvn
test -Dtest=com.kylinolap.job.BuildCubeWithEngineTest -DfailIfNoTests=false</code></pre></div>
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">mvn
test -Dtest=org.apache.kylin.job.BuildCubeWithEngineTest -DfailIfNoTests=false -P sandbox
+mvn test -Dtest=org.apache.kylin.job.BuildIIWithEngineTest -DfailIfNoTests=false -P sandbox</code></pre></div>
 
 <p>Run other tests, the end-to-end cube building test is exclueded</p>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">mvn
test</code></pre></div>
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">mvn
test -fae -P sandbox</code></pre></div>
 
 <h3 id="launch-kylin-web-server">Launch Kylin Web Server</h3>
 
-<p>In your Eclipse IDE, launch <code class="highlighter-rouge">com.kylinolap.rest.DebugTomcat</code>
with specifying VM arguments “-Dspring.profiles.active=sandbox”. (By default Kylin
server will listen on 7070 port; If you want to use another port, please specify it as a parameter
when run `DebugTomcat)</p>
+<p>Copy server/src/main/webapp/WEB-INF to webapp/app/WEB-INF</p>
+
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">cp
-r server/src/main/webapp/WEB-INF webapp/app/WEB-INF</code></pre></div>
+
+<p>In IDE, launch <code class="highlighter-rouge">org.apache.kylin.rest.DebugTomcat</code>
with working directory set to the /server folder. (By default Kylin server will listen on
7070 port; If you want to use another port, please specify it as a parameter when run `DebugTomcat)</p>
 
 <p>Check Kylin Web available at http://localhost:7070/kylin (user:ADMIN,password:KYLIN)</p>
 
+<p>For IntelliJ IDEA users, need modify “server/kylin-server.iml” file, replace
all “PROVIDED” to “COMPILE”, otherwise an “java.lang.NoClassDefFoundError:
org/apache/catalina/LifecycleListener” error may be thrown;</p>
 
 						</div>
 					</div>

Modified: incubator/kylin/site/docs/development/metadata_store.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/docs/development/metadata_store.html?rev=1692810&r1=1692809&r2=1692810&view=diff
==============================================================================
--- incubator/kylin/site/docs/development/metadata_store.html (original)
+++ incubator/kylin/site/docs/development/metadata_store.html Mon Jul 27 05:37:41 2015
@@ -2148,33 +2148,48 @@
 
 							<p>Kylin organizes all of its metadata(including cube descriptions and instances,
projects, inverted index description and instances, jobs, tables and dictionaries) as a hierarchy
file system. However, Kylin uses hbase to store it, rather than normal file system. If you
check your kylin configuration file(kylin.properties) you will find such a line:</p>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">#
The metadata store in hbase
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">##
The metadata store in hbase
 kylin.metadata.url=kylin_metadata_qa@hbase:sandbox.hortonworks.com:2181:/hbase-unsecure</code></pre></div>
 
 <p>This indicates that the metadata will be saved as a htable called <code class="highlighter-rouge">kylin_metadata_qa</code>.
You can scan the htable in hbase shell to check it out.</p>
 
-<h1 id="backup-metadata-store">Backup Metadata Store</h1>
+<h2 id="backup-metadata-store-with-binary-package">Backup Metadata Store with binary
package</h2>
 
 <p>Sometimes you need to backup the Kylin’s Metadata Store from hbase to your
disk file system.<br />
-In such cases, assuming you’re on the hadoop CLI(or sandbox) where you deployed Kylin,
you can use:</p>
+In such cases, assuming you’re on the hadoop CLI(or sandbox) where you deployed Kylin,
you can go to KYLIN_HOME and run :</p>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">mkdir
~/meta_dump
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">bin/metastore.sh
backup</code></pre></div>
 
-hbase  org.apache.hadoop.util.RunJar  PATH_TO_KYLIN_JOB_JAR_FOLDER/kylin-job-x.x.x-SNAPSHOT-job.jar
com.kylinolap.common.persistence.ResourceTool  copy PATH_TO_KYLIN_CONFIG/kylin.properties
~/meta_dump</code></pre></div>
+<p>to dump your metadata to your local folder a folder under KYLIN_HOME/metadata_backps,
the folder is named after current time with the syntax: KYLIN_HOME/meta_backups/meta_year_month_day_hour_minute_second</p>
 
-<p>to dump your metadata to your local folder ~/meta_dump.</p>
-
-<h1 id="restore-metadata-store">Restore Metadata Store</h1>
+<h2 id="restore-metadata-store-with-binary-package">Restore Metadata Store with binary
package</h2>
 
 <p>In case you find your metadata store messed up, and you want to restore to a previous
backup:</p>
 
-<p>first clean up the metadata store:</p>
+<p>Firstly, reset the metadata store (this will clean everything of the Kylin metadata
store in hbase, make sure to backup):</p>
+
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">bin/metastore.sh
reset</code></pre></div>
+
+<p>Then upload the backup metadata to Kylin’s metadata store:</p>
+
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">bin/metastore.sh
restore $KYLIN_HOME/meta_backups/meta_xxxx_xx_xx_xx_xx_xx</code></pre></div>
+
+<h2 id="backuprestore-metadata-in-development-env-available-since-073">Backup/Restore
metadata in development env (available since 0.7.3)</h2>
+
+<p>When developing/debugging Kylin, typically you have a dev machine with an IDE, and
a backend sandbox. Usually you’ll write code and run test cases at dev machine. It would
be troublesome if you always have to put a binary package in the sandbox to check the metadata.
There is a helper class called SandboxMetastoreCLI to help you download/upload metadata locally
at your dev machine. Follow the Usage information and run it in your IDE.</p>
+
+<h2 id="cleanup-unused-resources-from-metadata-store-available-since-073">Cleanup unused
resources from Metadata Store (available since 0.7.3)</h2>
+<p>As time goes on, some resources like dictionary, table snapshots became useless
(as the cube segment be dropped or merged), but they still take space there; You can run command
to find and cleanup them from metadata store:</p>
+
+<p>Firstly, run a check, this is safe as it will not change anything:</p>
+
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">bin/metastore.sh
clean</code></pre></div>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">hbase
 org.apache.hadoop.util.RunJar PATH_TO_KYLIN_JOB_JAR_FOLDER/kylin-job-x.x.x-SNAPSHOT-job.jar
com.kylinolap.common.persistence.ResourceTool  reset</code></pre></div>
+<p>The resources that will be dropped will be listed;</p>
 
-<p>then upload the backup metadata in ~/meta_dump to Kylin’s metadata store:</p>
+<p>Next, add the “–delete true” parameter to cleanup those resources;
before this, make sure you have made a backup of the metadata store;</p>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">hbase
 org.apache.hadoop.util.RunJar  PATH_TO_KYLIN_JOB_JAR_FOLDER/kylin-job-x.x.x-SNAPSHOT-job.jar
com.kylinolap.common.persistence.ResourceTool  copy ~/meta_dump PATH_TO_KYLIN_CONFIG/kylin.properties</code></pre></div>
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">bin/metastore.sh
clean --delete true</code></pre></div>
 
 
 						</div>

Modified: incubator/kylin/site/docs/howto/howto_docs.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/docs/howto/howto_docs.html?rev=1692810&r1=1692809&r2=1692810&view=diff
==============================================================================
--- incubator/kylin/site/docs/howto/howto_docs.html (original)
+++ incubator/kylin/site/docs/howto/howto_docs.html Mon Jul 27 05:37:41 2015
@@ -2262,7 +2262,7 @@ Then access http://127.0.0.1:4000 in you
 <h3 id="setup">Setup</h3>
 
 <ol>
-  <li><code class="highlighter-rouge">cd docs/website</code></li>
+  <li><code class="highlighter-rouge">cd website</code></li>
   <li><code class="highlighter-rouge">svn co https://svn.apache.org/repos/asf/incubator/kylin/site
_site</code></li>
   <li><code class="highlighter-rouge">sudo apt-get install rubygems ruby2.1-dev
zlib1g-dev</code> (linux)</li>
   <li><code class="highlighter-rouge">sudo gem install bundler github-pages jekyll</code></li>

Modified: incubator/kylin/site/docs/install/kylin_docker.html
URL: http://svn.apache.org/viewvc/incubator/kylin/site/docs/install/kylin_docker.html?rev=1692810&r1=1692809&r2=1692810&view=diff
==============================================================================
--- incubator/kylin/site/docs/install/kylin_docker.html (original)
+++ incubator/kylin/site/docs/install/kylin_docker.html Mon Jul 27 05:37:41 2015
@@ -2164,9 +2164,9 @@
 <div class="highlight"><pre><code class="language-groff" data-lang="groff">$
wget https://raw.githubusercontent.com/sequenceiq/docker-kylin/master/ambari-functions
  $ source ambari-functions</code></pre></div>
 
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">$
kylin-deploy-cluster 3</code></pre></div>
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">$
kylin-deploy-cluster 1</code></pre></div>
 
-<p>You can specify the number of nodes you’d like to have in your cluster (3 in
this case). Once we installed all the necessary Hadoop<br />
+<p>You can specify the number of nodes you’d like to have in your cluster (1 in
this case). Once we installed all the necessary Hadoop<br />
 services we’ll build Kylin on top of it and then you can reach the UI on:</p>
 
 <div class="highlight"><pre><code class="language-groff" data-lang="groff">#Ambari
Dashboard
@@ -2175,7 +2175,7 @@ http://&lt;container_ip&gt;:8080</code><
 <p>Use <code class="highlighter-rouge">admin/admin</code> to login. Make
sure HBase is running.</p>
 
 <div class="highlight"><pre><code class="language-groff" data-lang="groff">#Kylin
Dashboard
-http://&lt;container_ip&gt;:7070</code></pre></div>
+http://&lt;container_ip&gt;:7070/kylin</code></pre></div>
 
 <p>The default credentials to login are: <code class="highlighter-rouge">ADMIN:KYLIN</code>.
<br />
 The cluster is pre-populated with sample data and is ready to build cubes as shown <a
href="../tutorial/create_cube.html">here</a>.</p>

Modified: incubator/kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/incubator/kylin/site/feed.xml?rev=1692810&r1=1692809&r2=1692810&view=diff
==============================================================================
--- incubator/kylin/site/feed.xml (original)
+++ incubator/kylin/site/feed.xml Mon Jul 27 05:37:41 2015
@@ -19,8 +19,8 @@
     <description>Apache Kylin Home</description>
     <link>http://kylin.incubator.apache.org/</link>
     <atom:link href="http://kylin.incubator.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
-    <pubDate>Tue, 21 Jul 2015 01:43:02 -0700</pubDate>
-    <lastBuildDate>Tue, 21 Jul 2015 01:43:02 -0700</lastBuildDate>
+    <pubDate>Sun, 26 Jul 2015 22:33:56 -0700</pubDate>
+    <lastBuildDate>Sun, 26 Jul 2015 22:33:56 -0700</lastBuildDate>
     <generator>Jekyll v2.5.3</generator>
     
       <item>



Mime
View raw message