kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ianzeng <clayto...@163.com>
Subject Re: failed to build kylin cube
Date Fri, 18 Mar 2016 15:33:04 GMT
OK, May thx :-)

> 在 2016年3月18日,23:28,ShaoFeng Shi <shaofengshi@apache.org> 写道:
> 
> Thanks for the information; We will check this and try to fix it in next release, a JIRA
is created for tracking this: https://issues.apache.org/jira/browse/KYLIN-1507 <https://issues.apache.org/jira/browse/KYLIN-1507>
> 
> ianzeng, please try the workaround that Pierre mentioned, it should work definitely.

> 
> Thanks Pierre for the sharing, it does save our effort on investigate this issue.
> 
> 2016-03-18 23:10 GMT+08:00 Pierre Gunet-Caplain <pgc@deezer.com <mailto:pgc@deezer.com>>:
> Hi,
> 
> I also had this problem on a cloudera distribution. I found out there is a regex that
filters the dependencies to keep only a few of them (hive-exec, hive-metastore and hive-hcatalog-core
if I'm right), and the regex is filtering out the cloudera jars names.
> 
> So I used another way send the dependencies : in kylin.properties I added kylin.job.mr.lib.dir=/local/directory/kylin/dep
. In this folder I created symbolic links to the cloudera jars enumerated above. These jars
are shipped to tmpjars at execution, and it works.
> 
> I hope it help,
> 
> Pierre
> Data Architect at Deezer
> 
> 
> 
> 2016-03-18 15:51 GMT+01:00 ianzeng <claytonly@163.com <mailto:claytonly@163.com>>:
> the find-hive-decency result is
>  bin/find-hive-dependency.sh 
> 16/03/18 18:09:26 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local
no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you
are connecting to a remote metastore.
> 
> Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/jars/hive-common-0.13.1-cdh5.3.3.jar!/hive-log4j.properties
> HCAT_HOME not found, try to find hcatalog path from hadoop home
> hive dependency: /opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hive/conf:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-client.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jline-0.9.94.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jpam-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.23.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-beeline.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hwi.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/junit-4.11.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-cli.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-server.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-ant-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jersey-server-1.14.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-exec-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/servlet-api-2.5-20081211.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-testutils-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-commons-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-3.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/htrace-core.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-cli-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/avro.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/libfb303-0.9.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-testutils.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hwi-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.23-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/oro-2.0.8.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-httpclient-3.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/derby-10.10.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/groovy-all-2.1.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hbase-handler.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hamcrest-core-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-jdbc-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-rdbms-3.2.9.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jersey-servlet-1.14.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/libthrift-0.9.0-cdh5-2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-contrib.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-tree-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/zookeeper.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-jdbc.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-beeline-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-protocol.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-metastore.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-serde-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jta-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-service-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/mail-1.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-exec.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ST4-4.0.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/tempus-fugit-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-secure.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-all-7.6.0.v20120127.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-serde.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-metastore-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/stax-api-1.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-secure-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/velocity-1.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-contrib-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jsr305-1.3.9.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-core-3.2.10.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-common-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-scheduler-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/super-csv-2.2.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-lang3-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-util-6.1.26.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ant-1.9.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-6.1.26.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.3.jar
>> 在 2016年3月18日,22:33,ShaoFeng Shi <shaofengshi@apache.org <mailto:shaofengshi@apache.org>>
写道:
>> 
>> Run $KYLIN_HOME/bin/find-hive-dependency.sh to see whether it can locate the HCat
jar files in the machine that running Kylin, please paste the output here for trouble shotting.

>> 
>> 2016-03-18 18:28 GMT+08:00 ianzeng <claytonly@163.com <mailto:claytonly@163.com>>:
>> HI all,
>> 
>> 	I has installed kylin 1.5 on redhead 6.3. I try build sample cube. But got error
msg as follow:
>> 
>> 2016-03-18 18:18:43,084 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an
attempt to override final parameter: hadoop.ssl.server.conf;  Ignoring.
>> 2016-03-18 18:18:43,093 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an
attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
>> 2016-03-18 18:18:43,509 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
session.id <http://session.id/> is deprecated. Instead, use dfs.metrics.session-id
>> 2016-03-18 18:18:43,921 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter:
File Output Committer Algorithm version is 1
>> 2016-03-18 18:18:43,933 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree
: [ ]
>> 2016-03-18 18:18:44,120 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception
running child : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat
not found
>> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2047)
>> 	at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
>> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:746)
>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at javax.security.auth.Subject.doAs(Subject.java:415)
>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>> Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat
not found
>> 	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953)
>> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2045)
>> 	... 8 more
>> 
>> 
>> What should I do to fix this problem ?
>> 
>> 
>> 
>> -- 
>> Best regards,
>> 
>> Shaofeng Shi
>> 
> 
> 
> 
> 
> 
> -- 
> Best regards,
> 
> Shaofeng Shi
> 


Mime
View raw message