carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sraghunandan <...@git.apache.org>
Subject [GitHub] incubator-carbondata pull request #611: [CARBONDATA-731] Enhance and correct...
Date Sat, 04 Mar 2017 17:13:16 GMT
Github user sraghunandan commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/611#discussion_r104292379
  
    --- Diff: docs/installation-guide.md ---
    @@ -92,77 +96,87 @@ To get started with CarbonData : [Quick Start](quick-start-guide.md),
[DDL Opera
     
        The following steps are only for Driver Nodes. (Driver nodes are the one which starts
the spark context.)
     
    -* [Build the CarbonData](https://cwiki.apache.org/confluence/display/CARBONDATA/Building+CarbonData+And+IDE+Configuration)
project and get the assembly jar from "./assembly/target/scala-2.10/carbondata_xxx.jar" and
put in the ``"<SPARK_HOME>/carbonlib"`` folder.
    +1. [Build the CarbonData](https://github.com/apache/incubator-carbondata/blob/master/build/README.md)
project and get the assembly jar from `./assembly/target/scala-2.1x/carbondata_xxx.jar` and
copy to `<SPARK_HOME>/carbonlib` folder.
    +
    +    **NOTE**: Create the carbonlib folder if it does not exists inside `<SPARK_HOME>`
path.
     
    -      NOTE: Create the carbonlib folder if it does not exists inside ``"<SPARK_HOME>"``
path.
    +2. Copy the `./processing/carbonplugins` folder from CarbonData repository to `<SPARK_HOME>/carbonlib/`
folder.
     
    -* Copy "carbonplugins" folder to ``"<SPARK_HOME>/carbonlib"`` folder from "./processing/"
folder of CarbonData repository.
    -      carbonplugins will contain .kettle folder.
    +    **NOTE**: carbonplugins will contain .kettle folder.
     
    -* Copy the "carbon.properties.template" to ``"<SPARK_HOME>/conf/carbon.properties"``
folder from conf folder of CarbonData repository.
    -* Modify the parameters in "spark-default.conf" located in the ``"<SPARK_HOME>/conf``"
    +3. Copy the `./conf/carbon.properties.template` file from CarbonData repository to `<SPARK_HOME>/conf/`
folder and rename the file to `carbon.properties`.
    +
    +4. Create `tar,gz` file of carbonlib folder and move it inside the carbonlib folder.
    +
    +```
    +	cd <SPARK_HOME>
    +	tar -zcvf carbondata.tar.gz carbonlib/
    +	mv carbondata.tar.gz carbonlib/
    +```
    +
    +5. Configure the properties mentioned in the following table in `<SPARK_HOME>/conf/spark-defaults.conf`
file.
     
     | Property | Description | Value |
     |---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
    -| spark.master | Set this value to run the Spark in yarn cluster mode. | Set "yarn-client"
to run the Spark in yarn cluster mode. |
    -| spark.yarn.dist.files | Comma-separated list of files to be placed in the working directory
of each executor. |``"<YOUR_SPARK_HOME_PATH>"/conf/carbon.properties`` |
    -| spark.yarn.dist.archives | Comma-separated list of archives to be extracted into the
working directory of each executor. |``"<YOUR_SPARK_HOME_PATH>"/carbonlib/carbondata_xxx.jar``
|
    -| spark.executor.extraJavaOptions | A string of extra JVM options to pass to executors.
For instance  NOTE: You can enter multiple values separated by space. |``-Dcarbon.properties.filepath="<YOUR_SPARK_HOME_PATH>"/conf/carbon.properties``
|
    -| spark.executor.extraClassPath | Extra classpath entries to prepend to the classpath
of executors. NOTE: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and append
the values in below parameter spark.driver.extraClassPath |``"<YOUR_SPARK_HOME_PATH>"/carbonlib/carbonlib/carbondata_xxx.jar``
|
    -| spark.driver.extraClassPath | Extra classpath entries to prepend to the classpath of
the driver. NOTE: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and append
the value in below parameter spark.driver.extraClassPath. |``"<YOUR_SPARK_HOME_PATH>"/carbonlib/carbonlib/carbondata_xxx.jar``
|
    -| spark.driver.extraJavaOptions | A string of extra JVM options to pass to the driver.
For instance, GC settings or other logging. |``-Dcarbon.properties.filepath="<YOUR_SPARK_HOME_PATH>"/conf/carbon.properties``
|
    -| carbon.kettle.home | Path that will be used by CarbonData internally to create graph
for loading the data. |``"<YOUR_SPARK_HOME_PATH>"/carbonlib/carbonplugins`` |
    +| spark.master | Set this value to run the Spark in yarn cluster mode. | Set yarn-client
to run the Spark in yarn cluster mode. |
    +| spark.yarn.dist.files | Comma-separated list of files to be placed in the working directory
of each executor. |"<SPARK_HOME>"/conf/carbon.properties |
    +| spark.yarn.dist.archives | Comma-separated list of archives to be extracted into the
working directory of each executor. |"<SPARK_HOME>"/carbonlib/carbondata.tar.gz |
    +| spark.executor.extraJavaOptions | A string of extra JVM options to pass to executors.
For instance  **NOTE**: You can enter multiple values separated by space. |-Dcarbon.properties.filepath=carbon.properties
|
    +| spark.executor.extraClassPath | Extra classpath entries to prepend to the classpath
of executors. **NOTE**: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and
append the values in below parameter spark.driver.extraClassPath |carbondata.tar.gz/carbonlib/*
|
    +| spark.driver.extraClassPath | Extra classpath entries to prepend to the classpath of
the driver. **NOTE**: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and append
the value in below parameter spark.driver.extraClassPath. |"<SPARK_HOME>"/carbonlib/carbonlib/*
|
    +| spark.driver.extraJavaOptions | A string of extra JVM options to pass to the driver.
For instance, GC settings or other logging. |-Dcarbon.properties.filepath="<SPARK_HOME>"/conf/carbon.properties
|
    +
     
    -* Add the following properties in ``<SPARK_HOME>/conf/ carbon.properties``:
    +6. Add the following properties in `<SPARK_HOME>/conf/carbon.properties`:
     
     | Property | Required | Description | Example | Default Value |
     |----------------------|----------|----------------------------------------------------------------------------------------|-------------------------------------|---------------|
     | carbon.storelocation | NO | Location where CarbonData will create the store and write
the data in its own format. | hdfs://HOSTNAME:PORT/Opt/CarbonStore | Propose to set HDFS directory|
    -| carbon.kettle.home | YES | Path that will be used by CarbonData internally to create
graph for loading the data. | $SPARK_HOME/carbonlib/carbonplugins |  |
    +| carbon.kettle.home | YES | Path that will be used by CarbonData internally to create
graph for loading the data. | carbondata.tar.gz/carbonlib/carbonplugins |  |
     
     
    -* Verify the installation.
    +7. Verify the installation.
     
     ```
          ./bin/spark-shell --master yarn-client --driver-memory 1g
          --executor-cores 2 --executor-memory 2G
     ```
    -  NOTE: Make sure you have permissions for CarbonData JARs and files through which driver
and executor will start.
    +  **NOTE**: Make sure you have permissions for CarbonData JARs and files through which
driver and executor will start.
     
       Getting started with CarbonData : [Quick Start](quick-start-guide.md), [DDL Operations
on CarbonData](ddl-operation-on-carbondata.md)
     
     ## Query Execution Using CarbonData Thrift Server
     
    -### Starting CarbonData Thrift Server
    +### Starting CarbonData Thrift Server.
     
    -   a. cd ``<SPARK_HOME>``
    +   a. cd `<SPARK_HOME>`
     
        b. Run the following command to start the CarbonData thrift server.
     
     ```
     ./bin/spark-submit --conf spark.sql.hive.thriftServer.singleSession=true
     --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer
    -$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
    +<SPARK_HOME>/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path>
     ```
     
     | Parameter | Description | Example |
     |---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|
    -| CARBON_ASSEMBLY_JAR | CarbonData assembly jar name present in the ``"<SPARK_HOME>"/carbonlib/``
folder. | carbondata_2.10-0.1.0-incubating-SNAPSHOT-shade-hadoop2.7.2.jar |
    -| carbon_store_path | This is a parameter to the CarbonThriftServer class. This a HDFS
path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation
parameter of carbon.properties. | ``hdfs//<host_name>:54310/user/hive/warehouse/carbon.store``
|
    +| CARBON_ASSEMBLY_JAR | CarbonData assembly jar name present in the "<SPARK_HOME>"/carbonlib/
folder. | carbondata_2.10-0.1.0-incubating-SNAPSHOT-shade-hadoop2.7.2.jar |
    +| carbon_store_path | This is a parameter to the CarbonThriftServer class. This a HDFS
path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation
parameter of carbon.properties. | hdfs//<host_name>:port/user/hive/warehouse/carbon.store
|
    --- End diff --
    
    fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message