kylin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From luke...@apache.org
Subject [5/6] incubator-kylin git commit: move pages to development menu for dev stuff
Date Tue, 04 Aug 2015 12:17:20 GMT
move pages to development menu for dev stuff


Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/a880dd4f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/a880dd4f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/a880dd4f

Branch: refs/heads/0.7-staging
Commit: a880dd4fb6e4443d2bcad0d57db3ae09bac65c4d
Parents: 9acfeb4
Author: Luke Han <lukehan@apache.org>
Authored: Tue Aug 4 19:32:35 2015 +0800
Committer: Luke Han <lukehan@apache.org>
Committed: Tue Aug 4 19:32:35 2015 +0800

----------------------------------------------------------------------
 website/_config.yml                           |    4 +-
 website/_data/development.yml                 |   30 +
 website/_data/docs.yml                        |   35 +-
 website/_dev/build_api.md                     |   56 +
 website/_dev/dev_env.md                       |  123 ++
 website/_dev/index.md                         |   17 +
 website/_dev/javascript_api.md                |   47 +
 website/_dev/metadata_store.md                |   61 +
 website/_dev/new_metadata.md                  |   30 +
 website/_dev/rest_api.md                      | 2031 +++++++++++++++++++
 website/_dev/test_minicluster.md              |   22 +
 website/_dev/web_tech.md                      |   46 +
 website/_docs/development/build_api.md        |   58 -
 website/_docs/development/dev_env.md          |  125 --
 website/_docs/development/javascript_api.md   |   49 -
 website/_docs/development/metadata_store.md   |   63 -
 website/_docs/development/new_metadata.md     |   32 -
 website/_docs/development/rest_api.md         | 2033 --------------------
 website/_docs/development/test_minicluster.md |   24 -
 website/_docs/development/web_tech.md         |   48 -
 website/_includes/dev_nav.html                |   33 +
 website/_includes/dev_ul.html                 |   26 +
 website/_layouts/dev.html                     |   44 +
 website/development/index.md                  |   18 -
 24 files changed, 2586 insertions(+), 2469 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_config.yml
----------------------------------------------------------------------
diff --git a/website/_config.yml b/website/_config.yml
index fe5afde..60ca55d 100644
--- a/website/_config.yml
+++ b/website/_config.yml
@@ -27,7 +27,7 @@ encoding: UTF-8
 timezone: America/Dawson 
 
 exclude: ["README.md", "Rakefile", "*.scss", "*.haml", "*.sh"]
-include: [_docs]
+include: [_docs,_dev]
 
 # Build settings
 markdown: kramdown
@@ -58,3 +58,5 @@ collections:
     output: true
   docs-cn:
     output: true    
+  dev:
+    output: true  

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_data/development.yml
----------------------------------------------------------------------
diff --git a/website/_data/development.yml b/website/_data/development.yml
new file mode 100644
index 0000000..217c0b7
--- /dev/null
+++ b/website/_data/development.yml
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Development menu items, for English one, dev-cn.yml is for Chinese one
+# The development menu is constructed in dev_nav.html with these data
+
+- title: Development Guide
+  dev:
+  - index
+  - dev_env
+  - metadata_store
+  - new_metadata
+  - rest_api
+  - build_api
+  - javascript_api
+  - web_tech
+  - test_minicluster
+  
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_data/docs.yml
----------------------------------------------------------------------
diff --git a/website/_data/docs.yml b/website/_data/docs.yml
index 8a6236d..7d03a64 100644
--- a/website/_data/docs.yml
+++ b/website/_data/docs.yml
@@ -1,3 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Docs menu items, for English one, docs-cn.yml is for Chinese one
+# The docs menu is constructed in docs_nav.html with these data
 - title: Getting Started
   docs:
   - index
@@ -34,21 +51,3 @@
   - howto/howto_upgrade
   - howto/howto_docs
   - howto/howto_release
-
-
-- title: Development
-  docs:
-  - development/dev_env
-  - development/metadata_store
-  - development/new_metadata
-  - development/rest_api
-  - development/build_api
-  - development/javascript_api
-  - development/web_tech
-  - development/test_minicluster
-  
-
-
-- title: Meta
-  docs:
-  - history
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_dev/build_api.md
----------------------------------------------------------------------
diff --git a/website/_dev/build_api.md b/website/_dev/build_api.md
new file mode 100644
index 0000000..62c7d1d
--- /dev/null
+++ b/website/_dev/build_api.md
@@ -0,0 +1,56 @@
+---
+layout: dev
+title:  "Build Cube with Restful API"
+categories: development
+permalink: /development/build_api.html
+---
+
+### 1.	Authentication
+*   Currently, Kylin uses [basic authentication](http://en.wikipedia.org/wiki/Basic_access_authentication).
+*   Add `Authorization` header to first request for authentication
+*   Or you can do a specific request by `POST http://localhost:7070/kylin/api/user/authentication`
+*   Once authenticated, client can go subsequent requests with cookies.
+*   Example
+{% highlight Groff markup %}
+    POST http://localhost:7070/kylin/api/user/authentication
+    
+    Authorization:Basic xxxxJD124xxxGFxxxSDF
+    Content-Type: application/json;charset=UTF-8
+{% endhighlight %}
+
+### 2.	Get details of cube. 
+*   `GET http://localhost:7070/kylin/api/cubes?cubeName={cube_name}&limit=15&offset=0`
+*   Client can find cube segment date ranges in returned cube detail.
+*   Example
+{% highlight Groff markup %}
+    GET http://localhost:7070/kylin/api/cubes?cubeName=test_kylin_cube_with_slr&limit=15&offset=0
+
+    Authorization:Basic xxxxJD124xxxGFxxxSDF
+    Content-Type: application/json;charset=UTF-8
+{% endhighlight %}
+### 3.	Then submit a build job of the cube. 
+*   `PUT http://localhost:7070/kylin/api/cubes/{cube_name}/rebuild`
+*   For put request body detail please refer to [service doc](rest_api.html). 
+    *   `startTime` and `endTime` should be utc timestamp.
+    *   `buildType` can be `BUILD` ,`MERGE` or `REFRESH`. `BUILD` is for building a new segment,
`REFRESH` for refreshing an existing segment. `MERGE` is for merging multiple existing segments
into one bigger segment.
+*   This method will return a newly created job instance, in which the uuid is the identity
of job to track job status.
+*   Example
+{% highlight Groff markup %}
+    PUT http://localhost:7070/kylin/api/cubes/test_kylin_cube_with_slr/rebuild
+
+    Authorization:Basic xxxxJD124xxxGFxxxSDF
+    Content-Type: application/json;charset=UTF-8
+    
+    {
+    	"startTime": 0,
+    	"endTime": 1388563200000,
+    	"buildType": "BUILD"
+    }
+{% endhighlight %}
+
+### 4.	Track job status. 
+*   `GET http://localhost:7070/kylin/api/jobs/{job_uuid}`
+*   Returned `job_status` represents current status of job.
+
+### 5.	If the job failed, you can resume the job. 
+*   `PUT http://localhost:7070/kylin/api/jobs/{job_uuid}/resume`

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_dev/dev_env.md
----------------------------------------------------------------------
diff --git a/website/_dev/dev_env.md b/website/_dev/dev_env.md
new file mode 100644
index 0000000..fedeceb
--- /dev/null
+++ b/website/_dev/dev_env.md
@@ -0,0 +1,123 @@
+---
+layout: dev
+title:  Setup Development Env
+categories: development
+permalink: /development/dev_env.html
+---
+
+Developers want to run kylin test cases or applications at their development machine. 
+
+By following this tutorial, you will be able to build kylin test cubes by running a specific
test case, and you can further run other test cases against the cubes having been built.
+
+## Environment on the Hadoop CLI
+
+Off-Hadoop-CLI installation requires you having a hadoop CLI machine (or a hadoop sandbox)
as well as your local develop machine. To make things easier we strongly recommend you starting
with running Kylin on a hadoop sandbox, like <http://hortonworks.com/products/hortonworks-sandbox/>.
In the following tutorial we'll go with **Hortonworks Sandbox 2.2.4**. It is recommended that
you provide enough memory to your sandbox, 8G or more is preferred.
+
+### Start Hadoop
+
+In Hortonworks sandbox, ambari helps to launch hadoop:
+
+{% highlight Groff markup %}
+ambari-agent start
+ambari-server start
+{% endhighlight %}
+	
+With both command successfully run you can go to ambari home page at <http://yoursandboxip:8080>
(user:admin,password:admin) to check everything's status. By default ambari disables Hbase,
you'll need manually start the `Hbase` service.
+
+For other hadoop distribution, basically start the hadoop cluster, make sure HDFS, YARN,
Hive, HBase are running.
+
+
+## Environment on the dev machine
+
+
+### Install maven
+
+The latest maven can be found at <http://maven.apache.org/download.cgi>, we create
a symbolic so that `mvn` can be run anywhere.
+
+{% highlight Groff markup %}
+cd ~
+wget http://xenia.sote.hu/ftp/mirrors/www.apache.org/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
+tar -xzvf apache-maven-3.2.5-bin.tar.gz
+ln -s /root/apache-maven-3.2.5/bin/mvn /usr/bin/mvn
+{% endhighlight %}
+
+### Compile
+
+First clone the Kylin project to your local:
+
+{% highlight Groff markup %}
+	git clone https://github.com/apache/incubator-kylin.git
+{% endhighlight %}
+	
+Install Kylin artifacts to the maven repo
+
+{% highlight Groff markup %}
+	mvn clean install -DskipTests
+{% endhighlight %}
+
+### Modify local configuration
+
+Local configuration must be modified to point to your hadoop sandbox (or CLI) machine. 
+
+* In **examples/test_case_data/sandbox/kylin.properties**
+   * Find `sandbox` and replace with your hadoop hosts (if you're using HDP sandbox, this
can be skipped)
+   * Fine `kylin.job.run.as.remote.cmd` and change it to "true" (in code repository the default
is false, which assume running it on hadoop CLI)
+   * Find `kylin.job.remote.cli.username` and `kylin.job.remote.cli.password`, fill in the
user name and password used to login hadoop cluster for hadoop command execution; If you're
using HDP sandbox, the default username is "root" and password is "hadoop".
+
+* In **examples/test_case_data/sandbox**
+   * For each configuration xml file, find all occurrence of `sandbox` and replace with your
hadoop hosts; (if you're using HDP sandbox, this can be skipped)
+
+An alternative to the host replacement is updating your `hosts` file to resolve `sandbox`
and `sandbox.hortonworks.com` to the IP of your sandbox machine.
+
+### Run unit tests
+
+Run a end-to-end cube building test, these special test cases will populate some sample cubes
in your metadata store and build them ready.
+It might take a while (maybe one hour), please keep patient.
+ 
+{% highlight Groff markup %}
+
+mvn test -Dtest=org.apache.kylin.job.BuildCubeWithEngineTest -DfailIfNoTests=false -P sandbox
+mvn test -Dtest=org.apache.kylin.job.BuildIIWithEngineTest -DfailIfNoTests=false -P sandbox
+	
+{% endhighlight %}
+	
+Run other tests, the end-to-end cube building test is exclueded
+
+{% highlight Groff markup %}
+
+	mvn test -fae -P sandbox
+
+{% endhighlight %}
+
+### Launch Kylin Web Server
+
+Copy server/src/main/webapp/WEB-INF to webapp/app/WEB-INF 
+
+{% highlight Groff markup %}
+cp -r server/src/main/webapp/WEB-INF webapp/app/WEB-INF 
+{% endhighlight %}
+
+In IDE, launch `org.apache.kylin.rest.DebugTomcat` with working directory set to the /server
folder. (By default Kylin server will listen on 7070 port; If you want to use another port,
please specify it as a parameter when run `DebugTomcat)
+
+Check Kylin Web available at http://localhost:7070/kylin (user:ADMIN,password:KYLIN)
+
+For IntelliJ IDEA users, need modify "server/kylin-server.iml" file, replace all "PROVIDED"
to "COMPILE", otherwise an "java.lang.NoClassDefFoundError: org/apache/catalina/LifecycleListener"
error may be thrown;
+
+
+## Setup IDE code formatter
+
+In case you're writting code for Kylin, you should make sure that your code in expected formats.
+
+For Eclipse users, just format the code before committing the code.
+
+For intellij IDEA users, you have to do a few more steps:
+
+1. Install "Eclipse Code Formatter" and configure it as follows:
+
+	![Eclipse_Code_Formatter_Config](/images/develop/eclipse_code_formatter_config.png)
+
+2. Disable intellij IDEA's "Optimize imports on the fly"
+
+	![Disable_Optimize_On_The_Fly](/images/develop/disable_import_on_the_fly.png)
+
+3. Format the code before committing the code.

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_dev/index.md
----------------------------------------------------------------------
diff --git a/website/_dev/index.md b/website/_dev/index.md
new file mode 100644
index 0000000..e98599c
--- /dev/null
+++ b/website/_dev/index.md
@@ -0,0 +1,17 @@
+---
+layout: dev
+title: Development Quick Guide
+permalink: /development/index.html
+---
+
+### Source Code  
+Apache Kylin source code is version controlled using Git version control:
+Commits [Summary](https://git-wip-us.apache.org/repos/asf?p=incubator-kylin.git;a=summary)
 
+Source Repo: [git://git.apache.org/incubator-kylin.git](git://git.apache.org/incubator-kylin.git)
 
+Mirrored to Github: [https://github.com/apache/incubator-kylin](https://github.com/apache/incubator-kylin)
+
+### Issue Tracking  
+Track issues on the "Kylin" Project on the Apache JIRA ([browse](http://issues.apache.org/jira/browse/KYLIN)).
+
+### Setup Development Env  
+Refer to this link for how to setup dev env: [Setup Development Env](/docs/development/dev_env.html)
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_dev/javascript_api.md
----------------------------------------------------------------------
diff --git a/website/_dev/javascript_api.md b/website/_dev/javascript_api.md
new file mode 100644
index 0000000..fc76e4d
--- /dev/null
+++ b/website/_dev/javascript_api.md
@@ -0,0 +1,47 @@
+---
+layout: dev
+title:  How to consume Kylin REST API in javascript
+categories: development
+permalink: /development/javascript_api.html
+---
+Kypin securicty authorization is based on basic access authorization, so when you want to
use API in your javascript,you need to involve the authorization info in http headers.
+
+## Here is a demo show how to use Kylin Query API.
+```
+$.ajaxSetup({
+      headers: { 'Authorization': "Basic eWFu**********X***ZA==", 'Content-Type': 'application/json;charset=utf-8'
} // use your own authorization code here
+    });
+    var request = $.ajax({
+       url: "http://hostname/kylin/api/query",
+       type: "POST",
+       data: '{"sql":"select count(*) from SUMMARY;","offset":0,"limit":50000,"partial":true,"project":"test"}',
+       dataType: "json"
+    });
+    request.done(function( msg ) {
+       alert(msg);
+    }); 
+    request.fail(function( jqXHR, textStatus ) {
+       alert( "Request failed: " + textStatus );
+  });
+
+```
+
+
+For what is basic access authorization ,click here http://en.wikipedia.org/wiki/Basic_access_authentication
+
+## Keypoint:
+1. add basic access authorization info in heders.
+2. use right ajax type and right data synax
+
+How to generate your authorization code (download and import "jquery.base64.js" from https://github.com/yckart/jquery.base64.js)
+
+```
+var authorizationCode = $.base64('encode', 'NT_USERNAME' + ":" + 'NT_PASSWORD');
+ 
+$.ajaxSetup({
+   headers: { 
+    'Authorization': "Basic " + authorizationCode, 
+    'Content-Type': 'application/json;charset=utf-8' 
+   }
+});
+```

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_dev/metadata_store.md
----------------------------------------------------------------------
diff --git a/website/_dev/metadata_store.md b/website/_dev/metadata_store.md
new file mode 100644
index 0000000..199a40a
--- /dev/null
+++ b/website/_dev/metadata_store.md
@@ -0,0 +1,61 @@
+---
+layout: dev
+title:  "Kylin Metadata Store"
+categories: development
+permalink: /development/metadata_store.html
+---
+
+Kylin organizes all of its metadata(including cube descriptions and instances, projects,
inverted index description and instances, jobs, tables and dictionaries) as a hierarchy file
system. However, Kylin uses hbase to store it, rather than normal file system. If you check
your kylin configuration file(kylin.properties) you will find such a line:
+
+{% highlight Groff markup %}
+## The metadata store in hbase
+kylin.metadata.url=kylin_metadata_qa@hbase:sandbox.hortonworks.com:2181:/hbase-unsecure
+{% endhighlight %}
+
+
+This indicates that the metadata will be saved as a htable called `kylin_metadata_qa`. You
can scan the htable in hbase shell to check it out.
+
+## Backup Metadata Store with binary package
+
+Sometimes you need to backup the Kylin's Metadata Store from hbase to your disk file system.
+In such cases, assuming you're on the hadoop CLI(or sandbox) where you deployed Kylin, you
can go to KYLIN_HOME and run :
+
+{% highlight Groff markup %}
+bin/metastore.sh backup
+{% endhighlight %}
+
+to dump your metadata to your local folder a folder under KYLIN_HOME/metadata_backps, the
folder is named after current time with the syntax: KYLIN_HOME/meta_backups/meta_year_month_day_hour_minute_second
+
+## Restore Metadata Store with binary package
+
+In case you find your metadata store messed up, and you want to restore to a previous backup:
+
+Firstly, reset the metadata store (this will clean everything of the Kylin metadata store
in hbase, make sure to backup):
+
+{% highlight Groff markup %}
+bin/metastore.sh reset
+{% endhighlight %}
+
+Then upload the backup metadata to Kylin's metadata store:
+{% highlight Groff markup %}
+bin/metastore.sh restore $KYLIN_HOME/meta_backups/meta_xxxx_xx_xx_xx_xx_xx
+{% endhighlight %}
+
+## Backup/Restore metadata in development env (available since 0.7.3)
+
+When developing/debugging Kylin, typically you have a dev machine with an IDE, and a backend
sandbox. Usually you'll write code and run test cases at dev machine. It would be troublesome
if you always have to put a binary package in the sandbox to check the metadata. There is
a helper class called SandboxMetastoreCLI to help you download/upload metadata locally at
your dev machine. Follow the Usage information and run it in your IDE.
+
+## Cleanup unused resources from Metadata Store (available since 0.7.3)
+As time goes on, some resources like dictionary, table snapshots became useless (as the cube
segment be dropped or merged), but they still take space there; You can run command to find
and cleanup them from metadata store:
+
+Firstly, run a check, this is safe as it will not change anything:
+{% highlight Groff markup %}
+bin/metastore.sh clean
+{% endhighlight %}
+
+The resources that will be dropped will be listed;
+
+Next, add the "--delete true" parameter to cleanup those resources; before this, make sure
you have made a backup of the metadata store;
+{% highlight Groff markup %}
+bin/metastore.sh clean --delete true
+{% endhighlight %}

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a880dd4f/website/_dev/new_metadata.md
----------------------------------------------------------------------
diff --git a/website/_dev/new_metadata.md b/website/_dev/new_metadata.md
new file mode 100644
index 0000000..851ab85
--- /dev/null
+++ b/website/_dev/new_metadata.md
@@ -0,0 +1,30 @@
+---
+layout: dev
+title:  "New meta data model structure"
+categories: development
+permalink: /development/new_metadata.html
+---
+
+Kylin is doing a round code refactor which will introduce the following two changes on meta
data:
+
+* Abstract a "model_desc" layer from "cube_desc"
+
+Before define a cube, user will need firstly define a model ("model_desc"); The model defines
which is the fact table, which are lookup tables and how they be joined;
+
+With the model be defined, when user define a cube ("cube_desc"), he/she only need to specify
the table/column name for a dimension, as the join conditions have already been defined;
+
+This abstraction is to extend the meta data to fulfill non-cube queries (coming soon);
+
+* Support data tables from multiple hive databases;
+
+User has the case that tables are from multiple hive database, and the table name might be
the same; To support this case Kylin will use the database name + table name as the unique
name for tables; And user need to specify the database name (if it is not "default") in SQL
when query Kylin. 
+
+Here is a sample; the fact table "test_kylin_fact" is from default hive database, you don't
need to specify the db name; while lookup table is from "edw", you need use "edw.test_cal_dt"
in the query:
+
+{% highlight Groff markup %}
+select test_cal_dt.Week_Beg_Dt, sum(price) as c1, count(1) as c2 
+ from test_kylin_fact
+ inner JOIN edw.test_cal_dt as test_cal_dt
+ ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt 
+ where test_kylin_fact.lstg_format_name='ABIN' 
+{% endhighlight %}


Mime
View raw message