Return-Path: X-Original-To: apmail-kylin-commits-archive@minotaur.apache.org Delivered-To: apmail-kylin-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B0C5A1859C for ; Wed, 17 Feb 2016 18:33:02 +0000 (UTC) Received: (qmail 73048 invoked by uid 500); 17 Feb 2016 18:33:02 -0000 Delivered-To: apmail-kylin-commits-archive@kylin.apache.org Received: (qmail 73015 invoked by uid 500); 17 Feb 2016 18:33:02 -0000 Mailing-List: contact commits-help@kylin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kylin.apache.org Delivered-To: mailing list commits@kylin.apache.org Received: (qmail 73004 invoked by uid 99); 17 Feb 2016 18:33:02 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Feb 2016 18:33:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id EAEF2C0D80 for ; Wed, 17 Feb 2016 18:33:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.671 X-Spam-Level: ** X-Spam-Status: No, score=2.671 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, KAM_LINEPADDING=1.2, RP_MATCHES_RCVD=-0.329] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id OaOW5ZhvR3kA for ; Wed, 17 Feb 2016 18:32:50 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 2E77F6637C for ; Wed, 17 Feb 2016 13:48:57 +0000 (UTC) Received: from svn01-us-west.apache.org (svn.apache.org [10.41.0.6]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 387BFE09CC for ; Wed, 17 Feb 2016 13:48:56 +0000 (UTC) Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id 145213A116B for ; Wed, 17 Feb 2016 13:48:56 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r1730835 [4/13] - in /kylin/site: ./ development/ docs/ docs/install/ docs2/ docs2/gettingstarted/ docs2/howto/ docs2/install/ docs2/tutorial/ Date: Wed, 17 Feb 2016 13:48:55 -0000 To: commits@kylin.apache.org From: liyang@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20160217134856.145213A116B@svn01-us-west.apache.org> Added: kylin/site/docs2/howto/howto_jdbc.html URL: http://svn.apache.org/viewvc/kylin/site/docs2/howto/howto_jdbc.html?rev=1730835&view=auto ============================================================================== --- kylin/site/docs2/howto/howto_jdbc.html (added) +++ kylin/site/docs2/howto/howto_jdbc.html Wed Feb 17 13:48:54 2016 @@ -0,0 +1,1805 @@ + + + + + + + + + + + + Apache Kylin | How to Use kylin Remote JDBC Driver + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ +
+ +
+
+
+

How to Use kylin Remote JDBC Driver

+ +

version: v1.2, since: v0.7.1

+ +
+

Authentication

+ +
Build on kylin authentication restful service. Supported parameters:
+
    +
  • user : username
    • +
  • password : password
    • +
  • ssl: true/false. Default be false; If true, all the services call will use https.
    • +
+ +

Connection URL format:

+ +
jdbc:kylin://<hostname>:<port>/<kylin_project_name>
+ +
    +
  • If “ssl” = true, the “port” should be Kylin server’s HTTPS port;
    • +
  • If “port” is not specified, the driver will use default port: HTTP 80, HTTPS 443;
    • +
  • The “kylin_project_name” must be specified and user need ensure it exists in Kylin server;
    • +
+ +

1. Query with Statement

+ +
Driver driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
+
+Properties info = new Properties();
+info.put("user", "ADMIN");
+info.put("password", "KYLIN");
+Connection conn = driver.connect("jdbc:kylin://localhost:7070/kylin_project_name", info);
+Statement state = conn.createStatement();
+ResultSet resultSet = state.executeQuery("select * from test_table");
+
+while (resultSet.next()) {
+    assertEquals("foo", resultSet.getString(1));
+    assertEquals("bar", resultSet.getString(2));
+    assertEquals("tool", resultSet.getString(3));
+}
+ +

2. Query with PreparedStatement

+ +
Supported prepared statement parameters:
+
    +
  • setString
    • +
  • setInt
    • +
  • setShort
    • +
  • setLong
    • +
  • setFloat
    • +
  • setDouble
    • +
  • setBoolean
    • +
  • setByte
    • +
  • setDate
    • +
  • setTime
    • +
  • setTimestamp
    • +
+ +
Driver driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
+Properties info = new Properties();
+info.put("user", "ADMIN");
+info.put("password", "KYLIN");
+Connection conn = driver.connect("jdbc:kylin://localhost:7070/kylin_project_name", info);
+PreparedStatement state = conn.prepareStatement("select * from test_table where id=?");
+state.setInt(1, 10);
+ResultSet resultSet = state.executeQuery();
+
+while (resultSet.next()) {
+    assertEquals("foo", resultSet.getString(1));
+    assertEquals("bar", resultSet.getString(2));
+    assertEquals("tool", resultSet.getString(3));
+}
+ +

3. Get query result set metadata

+

Kylin jdbc driver supports metadata list methods:
+List catalog, schema, table and column with sql pattern filters(such as %).

+ +
Driver driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
+Properties info = new Properties();
+info.put("user", "ADMIN");
+info.put("password", "KYLIN");
+Connection conn = driver.connect("jdbc:kylin://localhost:7070/kylin_project_name", info);
+Statement state = conn.createStatement();
+ResultSet resultSet = state.executeQuery("select * from test_table");
+
+ResultSet tables = conn.getMetaData().getTables(null, null, "dummy", null);
+while (tables.next()) {
+    for (int i = 0; i < 10; i++) {
+        assertEquals("dummy", tables.getString(i + 1));
+    }
+}
+ + +
+
+
+
+
+
+ + +
+
+
+
+
+ + + + +
+ The contents of this website are © 2015 Apache Software Foundation under the terms of the Apache License v2 . Apache Kylin and + its logo are trademarks of the Apache Software Foundation. +
+ +
+
+
+ + +
+
+ +
+ + + + + + Added: kylin/site/docs2/howto/howto_ldap_and_sso.html URL: http://svn.apache.org/viewvc/kylin/site/docs2/howto/howto_ldap_and_sso.html?rev=1730835&view=auto ============================================================================== --- kylin/site/docs2/howto/howto_ldap_and_sso.html (added) +++ kylin/site/docs2/howto/howto_ldap_and_sso.html Wed Feb 17 13:48:54 2016 @@ -0,0 +1,1837 @@ + + + + + + + + + + + + Apache Kylin | How to Enable Security with LDAP and SSO + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ +
+ +
+
+
+

How to Enable Security with LDAP and SSO

+ +

version: v2.0, since: v1.0

+ +
+

Enable LDAP authentication

+ +

Kylin supports LDAP authentication for enterprise or production deployment; This is implemented with Spring Security framework; Before enable LDAP, please contact your LDAP administrator to get necessary information, like LDAP server URL, username/password, search patterns;

+ +

Configure LDAP server info

+ +

Firstly, provide LDAP URL, and username/password if the LDAP server is secured; The password in kylin.properties need be salted; You can Google “Generate a BCrypt Password” or run org.apache.kylin.rest.security.PasswordPlaceholderConfigurer to get a hash of your password.

+ +
ldap.server=ldap://<your_ldap_host>:<port>
+ldap.username=<your_user_name>
+ldap.password=<your_password_hash>
+
+
+ +

Secondly, provide the user search patterns, this is by LDAP design, here is just a sample:

+ +
ldap.user.searchBase=OU=UserAccounts,DC=mycompany,DC=com
+ldap.user.searchPattern=(&(AccountName={0})(memberOf=CN=MYCOMPANY-USERS,DC=mycompany,DC=com))
+ldap.user.groupSearchBase=OU=Group,DC=mycompany,DC=com
+
+
+ +

If you have service accounts (e.g, for system integration) which also need be authenticated, configure them in ldap.service.*; Otherwise, leave them be empty;

+ +

Configure the administrator group and default role

+ +

To map an LDAP group to the admin group in Kylin, need set the “acl.adminRole” to “ROLE_” + GROUP_NAME. For example, in LDAP the group “KYLIN-ADMIN-GROUP” is the list of administrators, here need set it as:

+ +
acl.adminRole=ROLE_KYLIN-ADMIN-GROUP
+acl.defaultRole=ROLE_ANALYST,ROLE_MODELER
+
+
+ +

The “acl.defaultRole” is a list of the default roles that grant to everyone, keep it as-is.

+ +

Enable LDAP

+ +

For Kylin v0.x and v1.x: set “kylin.sandbox=false” in conf/kylin.properties, then restart Kylin server;
+For Kylin since v2.0: set “kylin.security.profile=ldap” in conf/kylin.properties, then restart Kylin server;

+ +

Enable SSO authentication

+ +

From v2.0, Kylin provides SSO with SAML. The implementation is based on Spring Security SAML Extension. You can read this reference to get an overall understand.

+ +

Before trying this, you should have successfully enabled LDAP and managed users with it, as SSO server may only do authentication, Kylin need search LDAP to get the user’s detail information.

+ +

Generate IDP metadata xml

+

Contact your IDP (ID provider), asking to generate the SSO metadata file; Usually you need provide three piece of info:

+ +
    +
  1. Partner entity ID, which is an unique ID of your app, e.g,: https://host-name/kylin/saml/metadata
    2. +
  2. App callback endpoint, to which the SAML assertion be posted, it need be: https://host-name/kylin/saml/SSO
    3. +
  3. Public certificate of Kylin server, the SSO server will encrypt the message with it.
    4. +
+ +

Generate JKS keystore for Kylin

+

As Kylin need send encrypted message (signed with Kylin’s private key) to SSO server, a keystore (JKS) need be provided. There are a couple ways to generate the keystore, below is a sample.

+ +

Assume kylin.crt is the public certificate file, kylin.key is the private certificate file; firstly create a PKCS#12 file with openssl, then convert it to JKS with keytool:

+ +
$ openssl pkcs12 -export -in kylin.crt -inkey kylin.key -out kylin.p12
+Enter Export Password: <export_pwd>
+Verifying - Enter Export Password: <export_pwd>
+
+
+$ keytool -importkeystore -srckeystore kylin.p12 -srcstoretype PKCS12 -srcstorepass <export_pwd> -alias 1 -destkeystore samlKeystore.jks -destalias kylin -destkeypass changeit
+
+Enter destination keystore password:  changeit
+Re-enter new password: changeit
+
+
+ +

It will put the keys to “samlKeystore.jks” with alias “kylin”;

+ +

Enable Higher Ciphers

+ +

Make sure your environment is ready to handle higher level crypto keys, you may need to download Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files, copy local_policy.jar and US_export_policy.jar to $JAVA_HOME/jre/lib/security .

+ +

Deploy IDP xml file and keystore to Kylin

+ +

The IDP metadata and keystore file need be deployed in Kylin web app’s classpath in $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes

+ +
    +
  1. Name the IDP file to sso_metadata.xml and then copy to Kylin’s classpath;
    2. +
  2. Name the keystore as “samlKeystore.jks” and then copy to Kylin’s classpath;
    3. +
  3. If you use another alias or password, remember to update that kylinSecurity.xml accordingly:
    4. +
+ +
<!-- Central storage of cryptographic keys -->
+<bean id="keyManager" class="org.springframework.security.saml.key.JKSKeyManager">
+	<constructor-arg value="classpath:samlKeystore.jks"/>
+	<constructor-arg type="java.lang.String" value="changeit"/>
+	<constructor-arg>
+		<map>
+			<entry key="kylin" value="changeit"/>
+		</map>
+	</constructor-arg>
+	<constructor-arg type="java.lang.String" value="kylin"/>
+</bean>
+
+
+
+ +

Other configurations

+

In conf/kylin.properties, add the following properties with your server information:

+ +
saml.metadata.entityBaseURL=https://host-name/kylin
+saml.context.scheme=https
+saml.context.serverName=host-name
+saml.context.serverPort=443
+saml.context.contextPath=/kylin
+
+
+ +

Please note, Kylin assume in the SAML message there is a “email” attribute representing the login user, and the name before @ will be used to search LDAP.

+ +

Enable SSO

+

Set “kylin.security.profile=saml” in conf/kylin.properties, then restart Kylin server; After that, type a URL like “/kylin” or “/kylin/cubes” will redirect to SSO for login, and jump back after be authorized. While login with LDAP is still available, you can type “/kylin/login” to use original way. The Rest API (/kylin/api/*) still use LDAP + basic authentication, no impact.

+ + +
+
+
+
+
+
+ + +
+
+
+
+
+ + + + +
+ The contents of this website are © 2015 Apache Software Foundation under the terms of the Apache License v2 . Apache Kylin and + its logo are trademarks of the Apache Software Foundation. +
+ +
+
+
+ + +
+
+ +
+ + + + + + Added: kylin/site/docs2/howto/howto_optimize_cubes.html URL: http://svn.apache.org/viewvc/kylin/site/docs2/howto/howto_optimize_cubes.html?rev=1730835&view=auto ============================================================================== --- kylin/site/docs2/howto/howto_optimize_cubes.html (added) +++ kylin/site/docs2/howto/howto_optimize_cubes.html Wed Feb 17 13:48:54 2016 @@ -0,0 +1,1908 @@ + + + + + + + + + + + + Apache Kylin | How to Optimize Cubes + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + +
+ +
+ +
+
+
+

How to Optimize Cubes

+ +

version: v0.7.2, since: v0.7.1

+ +
+

Hierarchies:

+ +

Theoretically for N dimensions you’ll end up with 2^N dimension combinations. However for some group of dimensions there are no need to create so many combinations. For example, if you have three dimensions: continent, country, city (In hierarchies, the “bigger” dimension comes first). You will only need the following three combinations of group by when you do drill down analysis:

+ +

group by continent
+group by continent, country
+group by continent, country, city

+ +

In such cases the combination count is reduced from 2^3=8 to 3, which is a great optimization. The same goes for the YEAR,QUATER,MONTH,DATE case.

+ +

If we Donate the hierarchy dimension as H1,H2,H3, typical scenarios would be:

+ +

A. Hierarchies on lookup table

+ + + + + + + + + + + + +
Fact table(joins)Lookup Table
column1,column2,,,,,, FKPK,,H1,H2,H3,,,,
+ +
+ +

B. Hierarchies on fact table

+ + + + + + + + +
Fact table
column1,column2,,,H1,H2,H3,,,,,,,
+ +
+ +

There is a special case for scenario A, where PK on the lookup table is accidentally being part of the hierarchies. For example we have a calendar lookup table where cal_dt is the primary key:

+ +

A*. Hierarchies on lookup table over its primary key

+ + + + + + + + +
Lookup Table(Calendar)
cal_dt(PK), week_beg_dt, month_beg_dt, quarter_beg_dt,,,
+ +
+ +

For cases like A* what you need is another optimization called “Derived Columns”

+ +

Derived Columns:

+ +

Derived column is used when one or more dimensions (They must be dimension on lookup table, these columns are called “Derived”) can be deduced from another(Usually it is the corresponding FK, this is called the “host column”)

+ +

For example, suppose we have a lookup table where we join fact table and it with “where DimA = DimX”. Notice in Kylin, if you choose FK into a dimension, the corresponding PK will be automatically querable, without any extra cost. The secret is that since FK and PK are always identical, Kylin can apply filters/groupby on the FK first, and transparently replace them to PK. This indicates that if we want the DimA(FK), DimX(PK), DimB, DimC in our cube, we can safely choose DimA,DimB,DimC only.

+ + + + + + + + + + + + +
Fact table(joins)Lookup Table
column1,column2,,,,,, DimA(FK) DimX(PK),,DimB, DimC
+ +
+ +

Let’s say that DimA(the dimension representing FK/PK) has a special mapping to DimB:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
dimAdimBdimC
1a?
2b?
3c?
4a?
+ +

in this case, given a value in DimA, the value of DimB is determined, so we say dimB can be derived from DimA. When we build a cube that contains both DimA and DimB, we simple include DimA, and marking DimB as derived. Derived column(DimB) does not participant in cuboids generation:

+ +

original combinations:
+ABC,AB,AC,BC,A,B,C

+ +

combinations when driving B from A:
+AC,A,C

+ +

at Runtime, in case queries like “select count(*) from fact_table inner join looup1 group by looup1 .dimB”, it is expecting cuboid containing DimB to answer the query. However, DimB will appear in NONE of the cuboids due to derived optimization. In this case, we modify the execution plan to make it group by DimA(its host column) first, we’ll get intermediate answer like:

+ + + + + + + + + + + + + + + + + + + + + + +
DimAcount(*)
11
21
31
41
+ +

Afterwards, Kylin will replace DimA values with DimB values(since both of their values are in lookup table, Kylin can load the whole lookup table into memory and build a mapping for them), and the intermediate result becomes:

+ + + + + + + + + + + + + + + + + + + + + + +
DimBcount(*)
a1
b1
c1
a1
+ +

After this, the runtime SQL engine(calcite) will further aggregate the intermediate result to:

+ + + + + + + + + + + + + + + + + + +
DimBcount(*)
a2
b1
c1
+ +

this step happens at query runtime, this is what it means “at the cost of extra runtime aggregation”

+ +
+
+
+
+
+
+ + +
+
+
+
+
+ + + + +
+ The contents of this website are © 2015 Apache Software Foundation under the terms of the Apache License v2 . Apache Kylin and + its logo are trademarks of the Apache Software Foundation. +
+ +
+
+
+ + +
+
+ +
+ + + + + +