kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: failed to build kylin cube
Date Fri, 18 Mar 2016 15:28:05 GMT
Thanks for the information; We will check this and try to fix it in next
release, a JIRA is created for tracking this:
https://issues.apache.org/jira/browse/KYLIN-1507

ianzeng, please try the workaround that Pierre mentioned, it should work
definitely.

Thanks Pierre for the sharing, it does save our effort on investigate this
issue.

2016-03-18 23:10 GMT+08:00 Pierre Gunet-Caplain <pgc@deezer.com>:

> Hi,
>
> I also had this problem on a cloudera distribution. I found out there is a
> regex that filters the dependencies to keep only a few of them (hive-exec,
> hive-metastore and hive-hcatalog-core if I'm right), and the regex is
> filtering out the cloudera jars names.
>
> So I used another way send the dependencies : in kylin.properties I
> added kylin.job.mr.lib.dir=/local/directory/kylin/dep . In this folder I
> created symbolic links to the cloudera jars enumerated above. These jars
> are shipped to tmpjars at execution, and it works.
>
> I hope it help,
>
> Pierre
> Data Architect at Deezer
>
>
>
> 2016-03-18 15:51 GMT+01:00 ianzeng <claytonly@163.com>:
>
>> the find-hive-decency result is
>>  bin/find-hive-dependency.sh
>> 16/03/18 18:09:26 WARN conf.HiveConf: DEPRECATED: Configuration property
>> hive.metastore.local no longer has any effect. Make sure to provide a valid
>> value for hive.metastore.uris if you are connecting to a remote metastore.
>>
>> Logging initialized using configuration in
>> jar:file:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/jars/hive-common-0.13.1-cdh5.3.3.jar!/hive-log4j.properties
>> HCAT_HOME not found, try to find hcatalog path from hadoop home
>> hive dependency:
>> /opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hive/conf:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-client.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jline-0.9.94.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jpam-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.23.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-beeline.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hwi.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/junit-4.11.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-cli.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-server.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-ant-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jersey-server-1.14.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-exec-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/servlet-api-2.5-20081211.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-testutils-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-commons-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-3.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/htrace-core.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-cli-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/avro.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/libfb303-0.9.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-testutils.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hwi-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.23-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/oro-2.0.8.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-httpclient-3.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/derby-10.10.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/groovy-all-2.1.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hbase-handler.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hamcrest-core-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-jdbc-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-rdbms-3.2.9.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jersey-servlet-1.14.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/libthrift-0.9.0-cdh5-2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-contrib.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-tree-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/zookeeper.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-jdbc.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-beeline-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-protocol.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-metastore.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-serde-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jta-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-service-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/mail-1.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-exec.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ST4-4.0.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/tempus-fugit-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-secure.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-all-7.6.0.v20120127.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-serde.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-metastore-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/stax-api-1.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-secure-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/velocity-1.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-contrib-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jsr305-1.3.9.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-core-3.2.10.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-common-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-scheduler-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/super-csv-2.2.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-lang3-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-util-6.1.26.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ant-1.9.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-6.1.26.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.3.jar
>>
>> 在 2016年3月18日,22:33,ShaoFeng Shi <shaofengshi@apache.org> 写道:
>>
>> Run $KYLIN_HOME/bin/find-hive-dependency.sh to see whether it can locate
>> the HCat jar files in the machine that running Kylin, please paste the
>> output here for trouble shotting.
>>
>> 2016-03-18 18:28 GMT+08:00 ianzeng <claytonly@163.com>:
>>
>>> HI all,
>>>
>>>
>>> 	I has installed kylin 1.5 on redhead 6.3. I try build sample cube. But got error
msg as follow:
>>>
>>>
>>> 2016-03-18 18:18:43,084 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an
attempt to override final parameter: hadoop.ssl.server.conf;  Ignoring.
>>> 2016-03-18 18:18:43,093 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an
attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
>>> 2016-03-18 18:18:43,509 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
session.id is deprecated. Instead, use dfs.metrics.session-id
>>> 2016-03-18 18:18:43,921 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter:
File Output Committer Algorithm version is 1
>>> 2016-03-18 18:18:43,933 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree
: [ ]
>>> 2016-03-18 18:18:44,120 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception
running child : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat
not found
>>> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2047)
>>> 	at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
>>> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:746)
>>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>>> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>>> 	at java.security.AccessController.doPrivileged(Native Method)
>>> 	at javax.security.auth.Subject.doAs(Subject.java:415)
>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>>> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>>> Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat
not found
>>> 	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953)
>>> 	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2045)
>>> 	... 8 more
>>>
>>>
>>>
>>> What should I do to fix this problem ?
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>
>>
>


-- 
Best regards,

Shaofeng Shi

Mime
View raw message