kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billy Liu <billy...@apache.org>
Subject Re: Issues in building cubes of Apache Kylin
Date Thu, 08 Dec 2016 06:44:57 GMT
Seems you were not using any Hadoop distribution, such HDP or CDH, but
Apache Hadoop, Apache Hive and Apache HBase.

Could you try export HIVE_CONF to your hive config path first?


2016-12-08 13:53 GMT+08:00 Samirul Haque <sahaque@adobe.com>:

> Hi Billy,
>
>
>
> Thanks for your response.
>
>
>
> Output of the bin/find-dependency.sh is :
>
>
>
> KYLIN_HOME is set to /usr/local/kylin
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/usr/local/hive/lib/
> log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/usr/local/hadoop/
> share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/
> impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.
> Log4jLoggerFactory]
>
>
>
> Logging initialized using configuration in jar:file:/usr/local/hive/lib/
> hive-common-2.1.0.jar!/hive-log4j2.properties Async: true
>
> HIVE_CONF is set to: /home/hduser/bin:/home/hduser/
> .local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/
> bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/
> hadoop/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/
> local/hive/bin:/usr/local/kylin/bin:/usr/local/Hbase/
> bin:/usr/local/hadoop/bin:/usr/lib/jvm/java-8-openjdk-
> amd64/bin:/usr/local/hive/bin:/usr/local/kylin/bin:/usr/
> local/Hbase/bin:/usr/local/hadoop/bin:/usr/lib/jvm/java-
> 8-openjdk-amd64/bin:/usr/local/hive/bin:/usr/local/
> kylin/bin:/usr/local/Hbase/bin:/usr/local/hadoop/bin:/
> usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/local/hive/bin:
> /usr/local/kylin/bin:/usr/local/Hbase/bin:/usr/local/
> hadoop/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/
> local/hive/bin:/usr/local/kylin/bin:/usr/local/Hbase/
> bin:/usr/local/hadoop/bin:/usr/lib/jvm/java-8-openjdk-
> amd64/bin:/usr/local/hive/bin:/usr/local/kylin/bin:/usr/
> local/Hbase/bin:/usr/local/hive/conf, use it to locate hive
> configurations.
>
> HCAT_HOME is set to: /usr/local/hive/hcatalog, use it to find hcatalog
> path:
>
> hive dependency: /home/hduser/bin:/home/hduser/
> .local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/
> bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/
> hadoop/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/
> local/hive/bin:/usr/local/kylin/bin:/usr/local/Hbase/
> bin:/usr/local/hadoop/bin:/usr/lib/jvm/java-8-openjdk-
> amd64/bin:/usr/local/hive/bin:/usr/local/kylin/bin:/usr/
> local/Hbase/bin:/usr/local/hadoop/bin:/usr/lib/jvm/java-
> 8-openjdk-amd64/bin:/usr/local/hive/bin:/usr/local/
> kylin/bin:/usr/local/Hbase/bin:/usr/local/hadoop/bin:/
> usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/local/hive/bin:
> /usr/local/kylin/bin:/usr/local/Hbase/bin:/usr/local/
> hadoop/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/
> local/hive/bin:/usr/local/kylin/bin:/usr/local/Hbase/
> bin:/usr/local/hadoop/bin:/usr/lib/jvm/java-8-openjdk-
> amd64/bin:/usr/local/hive/bin:/usr/local/kylin/bin:/usr/
> local/Hbase/bin:/usr/local/hive/conf:/usr/local/hive/lib/
> antlr-runtime-3.4.jar:/usr/local/hive/lib/servlet-api-2.
> 4.jar:/usr/local/hive/lib/hive-llap-client-2.1.0.jar:/
> usr/local/hive/lib/twill-zookeeper-0.6.0-incubating.
> jar:/usr/local/hive/lib/bonecp-0.8.0.RELEASE.jar:/usr/
> local/hive/lib/hbase-annotations-1.1.1.jar:/usr/
> local/hive/lib/zookeeper-3.4.6.jar:/usr/local/hive/lib/
> guice-assistedinject-3.0.jar:/usr/local/hive/lib/ivy-2.4.0.
> jar:/usr/local/hive/lib/datanucleus-rdbms-4.1.7.jar:/
> usr/local/hive/lib/parquet-hadoop-bundle-1.8.1.jar:/usr/
> local/hive/lib/hive-exec-2.1.0.jar:/usr/local/hive/lib/
> hbase-common-1.1.1.jar:/usr/local/hive/lib/commons-lang3-
> 3.1.jar:/usr/local/hive/lib/commons-pool-1.5.4.jar:/usr/
> local/hive/lib/accumulo-start-1.6.0.jar:/usr/local/hive/lib/
> twill-common-0.6.0-incubating.jar:/usr/local/hive/lib/jetty-
> 6.1.26.jar:/usr/local/hive/lib/jackson-core-2.4.2.jar:/
> usr/local/hive/lib/commons-httpclient-3.0.1.jar:/usr/
> local/hive/lib/hive-jdbc-2.1.0.jar:/usr/local/hive/lib/
> hive-serde-2.1.0.jar:/usr/local/hive/lib/hbase-prefix-
> tree-1.1.1.jar:/usr/local/hive/lib/commons-lang-2.6.jar:
> /usr/local/hive/lib/opencsv-2.3.jar:/usr/local/hive/lib/jsp-
> api-2.1.jar:/usr/local/hive/lib/antlr-2.7.7.jar:/usr/
> local/hive/lib/libfb303-0.9.3.jar:/usr/local/hive/lib/
> slider-core-0.90.2-incubating.jar:/usr/local/hive/lib/jsp-2.
> 1-6.1.14.jar:/usr/local/hive/lib/metrics-json-3.1.0.jar:/
> usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar:/usr/
> local/hive/lib/jamon-runtime-2.3.1.jar:/usr/local/hive/lib/
> avro-1.7.7.jar:/usr/local/hive/lib/jetty-all-server-7.6.
> 0.v20120127.jar:/usr/local/hive/lib/activation-1.1.jar:/
> usr/local/hive/lib/derby.jar:/usr/local/hive/lib/stax-api-1.
> 0.1.jar:/usr/local/hive/lib/tephra-api-0.6.0.jar:/usr/
> local/hive/lib/netty-all-4.0.23.Final.jar:/usr/local/hive/
> lib/jersey-client-1.9.jar:/usr/local/hive/lib/metrics-
> core-3.1.0.jar:/usr/local/hive/lib/paranamer-2.3.jar:/
> usr/local/hive/lib/snappy-java-1.0.5.jar:/usr/local/
> hive/lib/metrics-core-2.2.0.jar:/usr/local/hive/lib/
> commons-codec-1.4.jar:/usr/local/hive/lib/commons-vfs2-2.
> 0.jar:/usr/local/hive/lib/javolution-5.5.1.jar:/usr/
> local/hive/lib/hive-cli-2.1.0.jar:/usr/local/hive/lib/hive-
> metastore-2.1.0.jar:/usr/local/hive/lib/hive-beeline-2.
> 1.0.jar:/usr/local/hive/lib/ant-1.9.1.jar:/usr/local/hive/
> lib/ant-1.6.5.jar:/usr/local/hive/lib/commons-cli-1.2.jar:/
> usr/local/hive/lib/mysql-connector-java-5.1.21.jar:/
> usr/local/hive/lib/jsr305-3.0.0.jar:/usr/local/hive/lib/
> commons-compress-1.9.jar:/usr/local/hive/lib/hive-llap-ext-
> client-2.1.0.jar:/usr/local/hive/lib/javax.jdo-3.2.0-m3.
> jar:/usr/local/hive/lib/disruptor-3.3.0.jar:/usr/
> local/hive/lib/jackson-xc-1.9.2.jar:/usr/local/hive/lib/jsp-
> api-2.1-6.1.14.jar:/usr/local/hive/lib/ant-launcher-1.9.1.
> jar:/usr/local/hive/lib/geronimo-jta_1.1_spec-1.1.1.
> jar:/usr/local/hive/lib/tempus-fugit-1.1.jar:/usr/
> local/hive/lib/hive-ant-2.1.0.jar:/usr/local/hive/lib/
> datanucleus-core-4.1.6.jar:/usr/local/hive/lib/hive-llap-
> common-2.1.0.jar:/usr/local/hive/lib/hive-shims-0.23-2.1.
> 0.jar:/usr/local/hive/lib/datanucleus-api-jdo-4.2.1.jar:
> /usr/local/hive/lib/guice-3.0.jar:/usr/local/hive/lib/
> findbugs-annotations-1.3.9-1.jar:/usr/local/hive/lib/
> jackson-jaxrs-1.9.2.jar:/usr/local/hive/lib/hive-accumulo-
> handler-2.1.0.jar:/usr/local/hive/lib/httpclient-4.4.jar:/
> usr/local/hive/lib/hbase-hadoop2-compat-1.1.1.jar:/usr/
> local/hive/lib/curator-framework-2.6.0.jar:/usr/
> local/hive/lib/aopalliance-1.0.jar:/usr/local/hive/lib/
> hive-testutils-2.1.0.jar:/usr/local/hive/lib/twill-api-0.6.
> 0-incubating.jar:/usr/local/hive/lib/dropwizard-metrics-
> hadoop-metrics2-reporter-0.1.0.jar:/usr/local/hive/lib/
> groovy-all-2.4.4.jar:/usr/local/hive/lib/metrics-jvm-3.
> 1.0.jar:/usr/local/hive/lib/log4j-api-2.4.1.jar:/usr/
> local/hive/lib/log4j-web-2.4.1.jar:/usr/local/hive/lib/
> jersey-server-1.14.jar:/usr/local/hive/lib/hbase-server-1.
> 1.1.jar:/usr/local/hive/lib/twill-core-0.6.0-incubating.
> jar:/usr/local/hive/lib/hive-llap-tez-2.1.0.jar:/usr/local/
> hive/lib/joda-time-2.5.jar:/usr/local/hive/lib/accumulo-
> core-1.6.0.jar:/usr/local/hive/lib/hive-hplsql-2.1.0.
> jar:/usr/local/hive/lib/jetty-sslengine-6.1.26.jar:/usr/
> local/hive/lib/hive-storage-api-2.1.0.jar:/usr/local/hive/
> lib/hive-llap-server-2.1.0.jar:/usr/local/hive/lib/json-
> 20090211.jar:/usr/local/hive/lib/antlr4-runtime-4.5.jar:/
> usr/local/hive/lib/jackson-annotations-2.4.0.jar:/usr/
> local/hive/lib/stringtemplate-3.2.1.jar:/usr/local/hive/lib/
> log4j-1.2-api-2.4.1.jar:/usr/local/hive/lib/regexp-1.3.jar:
> /usr/local/hive/lib/hive-service-2.1.0.jar:/usr/local/
> hive/lib/jline-2.12.jar:/usr/local/hive/lib/jasper-
> compiler-5.5.23.jar:/usr/local/hive/lib/asm-commons-3.
> 1.jar:/usr/local/hive/lib/jetty-all-7.6.0.v20120127.jar:
> /usr/local/hive/lib/guice-servlet-3.0.jar:/usr/local/
> hive/lib/jetty-util-6.1.26.jar:/usr/local/hive/lib/twill-
> discovery-core-0.6.0-incubating.jar:/usr/local/
> hive/lib/hbase-client-1.1.1.jar:/usr/local/hive/lib/
> plexus-utils-1.5.6.jar:/usr/local/hive/lib/servlet-api-2.
> 5-6.1.14.jar:/usr/local/hive/lib/maven-scm-api-1.4.jar:/
> usr/local/hive/lib/snappy-0.2.jar:/usr/local/hive/lib/maven-
> scm-provider-svn-commons-1.4.jar:/usr/local/hive/lib/jdo-
> api-3.0.1.jar:/usr/local/hive/lib/jta-1.1.jar:/usr/local/
> hive/lib/junit-4.11.jar:/usr/local/hive/lib/joni-2.1.2.jar:
> /usr/local/hive/lib/commons-math-2.2.jar:/usr/local/hive/
> lib/htrace-core-3.1.0-incubating.jar:/usr/local/
> hive/lib/javax.inject-1.jar:/usr/local/hive/lib/hive-
> service-rpc-2.1.0.jar:/usr/local/hive/lib/jackson-
> databind-2.4.2.jar:/usr/local/hive/lib/jsp-api-2.0.jar:/usr/
> local/hive/lib/hive-common-2.1.0.jar:/usr/local/hive/lib/
> transaction-api-1.1.jar:/usr/local/hive/lib/hive-hbase-
> handler-2.1.0.jar:/usr/local/hive/lib/eigenbase-properties-
> 1.1.5.jar:/usr/local/hive/lib/hbase-hadoop2-compat-1.1.1-
> tests.jar:/usr/local/hive/lib/fastutil-6.5.6.jar:/usr/local/
> hive/lib/jcommander-1.32.jar:/usr/local/hive/lib/httpcore-4.
> 4.jar:/usr/local/hive/lib/accumulo-fate-1.6.0.jar:/usr/
> local/hive/lib/hive-contrib-2.1.0.jar:/usr/local/hive/lib/
> gson-2.2.4.jar:/usr/local/hive/lib/commons-logging-1.2.
> jar:/usr/local/hive/lib/guava-14.0.1.jar:/usr/local/hive/
> lib/hbase-hadoop-compat-1.1.1.jar:/usr/local/hive/lib/javax.
> servlet-3.0.0.v201112011016.jar:/usr/local/hive/lib/hbase-
> common-1.1.1-tests.jar:/usr/local/hive/lib/hamcrest-core-
> 1.3.jar:/usr/local/hive/lib/asm-tree-3.1.jar:/usr/local/
> hive/lib/ST4-4.0.4.jar:/usr/local/hive/lib/hive-shims-2.1.
> 0.jar:/usr/local/hive/lib/jcodings-1.0.8.jar:/usr/local/
> hive/lib/commons-collections-3.2.2.jar:/usr/local/hive/lib/
> tephra-hbase-compat-1.0-0.6.0.jar:/usr/local/hive/lib/
> curator-client-2.6.0.jar:/usr/local/hive/lib/geronimo-
> annotation_1.0_spec-1.1.1.jar:/usr/local/hive/lib/hbase-
> procedure-1.1.1.jar:/usr/local/hive/lib/netty-3.7.0.
> Final.jar:/usr/local/hive/lib/commons-dbcp-1.4.jar:/usr/
> local/hive/lib/hive-orc-2.1.0.jar:/usr/local/hive/lib/
> tephra-core-0.6.0.jar:/usr/local/hive/lib/maven-scm-
> provider-svnexe-1.4.jar:/usr/local/hive/lib/accumulo-trace-
> 1.6.0.jar:/usr/local/hive/lib/org.abego.treelayout.core-1.0.
> 1.jar:/usr/local/hive/lib/velocity-1.5.jar:/usr/local/
> hive/lib/log4j-core-2.4.1.jar:/usr/local/hive/lib/pentaho-
> aggdesigner-algorithm-5.1.5-jhyde.jar:/usr/local/hive/lib/
> jasper-runtime-5.5.23.jar:/usr/local/hive/lib/protobuf-
> java-2.5.0.jar:/usr/local/hive/lib/commons-io-2.4.jar:/
> usr/local/hive/lib/curator-recipes-2.6.0.jar:/usr/local/
> hive/lib/super-csv-2.2.0.jar:/usr/local/hive/lib/hive-shims-
> common-2.1.0.jar:/usr/local/hive/lib/janino-2.7.6.jar:/
> usr/local/hive/lib/libthrift-0.9.3.jar:/usr/local/hive/lib/
> twill-discovery-api-0.6.0-incubating.jar:/usr/local/
> hive/lib/geronimo-jaspic_1.0_spec-1.0.jar:/usr/local/hive/
> lib/commons-el-1.0.jar:/usr/local/hive/lib/hbase-protocol-
> 1.1.1.jar:/usr/local/hive/lib/asm-3.1.jar:/usr/local/hive/
> lib/hive-hwi-2.1.0.jar:/usr/local/hive/lib/hive-shims-
> scheduler-2.1.0.jar:/usr/local/hive/lib/jpam-1.1.jar:/
> usr/local/hive/lib/commons-compiler-2.7.6.jar:/usr/local/
> hive/lib/mail-1.4.1.jar:/usr/local/hive/hcatalog/share/
> hcatalog/hive-hcatalog-core-2.1.0.jar
>
>
>
>
>
> Hadoop distribution which was used is :
>
> http://redrockdigimark.com/apachemirror/hadoop/core/
> stable/hadoop-2.7.3.tar.gz
>
>
>
> Please let me know if you need any other information.
>
>
>
> *From:* Billy Liu [mailto:billyliu@apache.org]
> *Sent:* Thursday, December 08, 2016 6:19 AM
> *To:* dev <dev@kylin.apache.org>
> *Cc:* dev@kylin.incubator.apache.org; Samirul Haque <sahaque@adobe.com>;
> Abhishek Shishodia <shishodi@adobe.com>
> *Subject:* Re: Issues in building cubes of Apache Kylin
>
>
>
> Hi Tarun,
>
>
>
> what's the output of bin/find-hive-dependency.sh? It seems something wrong
> in the environment variables.
>
> Which Hadoop distribution were you using?
>
>
>
> 2016-12-07 23:04 GMT+08:00 Tarun Vashisth <vashisth@adobe.com>:
>
> Hi,
>
>
>
> We are trying to build cubes for data and while doing so, we are getting
> the following error during the step 3 of cube creating
>
> File does not exist: hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/
> hduser341814501/.staging/job_local341814501_0007/libjars/
> hive-exec-2.1.0.jar
>
>
> Earlier, it was trying to find the jars at hdfs://localhost:54310/usr/local/hive/lib.
> We manually put all the jars of hive there and after that, it started
> failing at this step.
>
>   *   Is kylin trying to create this directory hduser../ and so on and
> unable to do so? Is it failing at that step?
>
> Hadoop Version: - 2.7.3
> Hive Version: - 2.1.0
> Hbase Version: - 1.2.4
> Kylin Version: - 1.6
>
> Any help would be much appreciated in this regard.
>
>
> Regards,
>
> Tarun
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message