Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B5147200BEF for ; Wed, 4 Jan 2017 15:51:35 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B39BE160B3A; Wed, 4 Jan 2017 14:51:35 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E1274160B4A for ; Wed, 4 Jan 2017 15:51:33 +0100 (CET) Received: (qmail 7784 invoked by uid 500); 4 Jan 2017 14:51:33 -0000 Mailing-List: contact commits-help@carbondata.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.incubator.apache.org Delivered-To: mailing list commits@carbondata.incubator.apache.org Received: (qmail 7775 invoked by uid 99); 4 Jan 2017 14:51:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2017 14:51:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8173EC002D for ; Wed, 4 Jan 2017 14:51:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -7.019 X-Spam-Level: X-Spam-Status: No, score=-7.019 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id doU8Hju7J1oj for ; Wed, 4 Jan 2017 14:51:11 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 1068F5FE18 for ; Wed, 4 Jan 2017 14:50:47 +0000 (UTC) Received: (qmail 2571 invoked by uid 99); 4 Jan 2017 14:50:47 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2017 14:50:47 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id EAD3FDFC11; Wed, 4 Jan 2017 14:50:46 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: chenliang613@apache.org To: commits@carbondata.incubator.apache.org Date: Wed, 04 Jan 2017 14:51:42 -0000 Message-Id: <2bc2838d70c648aa8d8a8f5f2772f5b1@git.apache.org> In-Reply-To: <07ee074769a242fcaf5e8fe86b3b051d@git.apache.org> References: <07ee074769a242fcaf5e8fe86b3b051d@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [58/69] incubator-carbondata-site git commit: Document Changes archived-at: Wed, 04 Jan 2017 14:51:35 -0000 http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/64ee1986/src/main/webapp/docs/latest/installation.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/docs/latest/installation.html b/src/main/webapp/docs/latest/installation.html index 05cbb5f..162d433 100644 --- a/src/main/webapp/docs/latest/installation.html +++ b/src/main/webapp/docs/latest/installation.html @@ -1,59 +1,4 @@ -Untitled Document.md -

Version: 0.2.0 | Last Published: 21-11-2016

- - Top -

Installation Guide

This tutorial will guide you through the installation and configuration of CarbonData in the following two modes:

This tutorial guides you through the installation and configuration of CarbonData in the following two modes:

Installing and Configuring CarbonData on Standalone Spark Cluster
Installing and Configuring CarbonData on “Spark on YARN” Cluster

Prerequisites

Hadoop HDFS and Yarn should be installed and running.
Spark should be installed and running in all the clients.
Spark should be installed and running on all the cluster nodes.
CarbonData user should have permission to access HDFS.

Procedure

The following steps are only for Driver Nodes.(Driver nodes are the one which starts the spark context.)

Build the CarbonData project and get the assembly jar from “./assembly/target/scala-2.10/carbondata_xxx.jar” and put in the “<SPARK_HOME>/carbonlib” folder.
-
(NOTE: Create the carbonlib folder if does not exists inside “<SPARK_HOME>” path.)
+
NOTE: Create the carbonlib folder if it does not exists inside “<SPARK_HOME>” path.
Add the carbonlib folder path in the Spark classpath. (Edit “<SPARK_HOME>/conf/spark-env.sh” file and modify the value of SPARK_CLASSPATH by appending “<SPARK_HOME>/carbonlib/*” to the existing value)
@@ -112,7 +51,7 @@ under the License.
Copy the “carbonplugins” folder to “<SPARK_HOME>/carbonlib” folder from “./processing/” folder of CarbonData repository.
-
(NOTE: carbonplugins will contain .kettle folder.)
+
NOTE: carbonplugins will contain .kettle folder.
In Spark node, configure the properties mentioned in the following table in “<SPARK_HOME>/conf/spark-defaults.conf” file.
@@ -160,7 +99,7 @@ under the License. NO Location where data CarbonData will create the store and write the data in its own format. hdfs://IP:PORT/Opt/CarbonStore -Propose +Propose to set HDFS directory carbon.kettle.home @@ -189,16 +128,17 @@ under the License.
CarbonData user should have permission to access HDFS.

Procedure

The following steps are only for Driver Nodes. (Driver nodes are the one which starts the spark context.)

Build the CarbonData project and get the assembly jar from “./assembly/target/scala-2.10/carbondata_xxx.jar” and put in the “<SPARK_HOME>/carbonlib” folder.
-
(NOTE: Create the carbonlib folder if does not exists inside “<SPARK_HOME>” path.)
+
NOTE: Create the carbonlib folder if it does not exists inside “<SPARK_HOME>” path.
-
Copy the carbon.properties.template to “<SPARK_HOME>/conf/carbon.properties” folder from “./conf/” of CarbonData repository. carbonplugins will contain .kettle folder.
+
Copy the "carbonplugins" folder to “<SPARK_HOME>/carbonlib” folder from “./processing/” of CarbonData repository. carbonplugins will contain .kettle folder.
-
Copy the “carbon.properties.template” to “<SPARK_HOME>/conf/carbon.properties” folder from conf folder of carbondata repository.
+
Copy the “carbon.properties.template” to “<SPARK_HOME>/conf/carbon.properties” folder from conf folder of CarbonData repository.
Modify the parameters in “spark-default.conf” located in the “<SPARK_HOME>/conf”
@@ -273,7 +213,7 @@ under the License. NO Location where data CarbonData will create the store and write the data in its own format. hdfs://IP:PORT/Opt/CarbonStore -Propose +Propose to set HDFS directory carbon.kettle.home @@ -321,14 +261,14 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path> carbon_store_path This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.proeprties. -hdfs//hacluster/user/hive/warehouse/carbon.storehdfs//10.10.10.10:54310 /user/hive/warehouse/carbon.store +hdfs//hacluster/user/hive/warehouse/carbon.store hdfs//10.10.10.10:54310 /user/hive/warehouse/carbon.store

Examples

Start with default memory and executors.

./bin/spark-submit --conf spark.sql.hive.thriftServer.singleSession=true --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer $SPARK_HOME/carbonlib/carbondata_2.10-0.1.0-incubating-SNAPSHOT-shade-hadoop2.7.2.jar hdfs://hacluster/user/hive/warehouse/carbon.store

@@ -338,7 +278,7 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>

./bin/spark-submit --conf spark.sql.hive.thriftServer.singleSession=true --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer --num-executors 3 --driver-memory 20g --executor-memory 250g --executor-cores 32 /srv/OSCON/BigData/HACluster/install/spark/sparkJdbc/lib/carbondata_2.10-0.1.0-incubating-SNAPSHOT-shade-hadoop2.7.2.jar hdfs://hacluster/user/hive/warehouse/carbon.store

Connecting to CarbonData Thrift Server Using Beeline

cd <SPARK_HOME>
 ./bin/beeline jdbc:hive2://<thrftserver_host>:port
@@ -346,7 +286,18 @@ $SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
 Example
 ./bin/beeline jdbc:hive2://10.10.10.10:10000

- -Top - + + +