kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaoxiang Yu <high.r...@icloud.com.INVALID>
Subject Re: regarding glue metastore
Date Tue, 16 Jul 2019 13:17:32 GMT
Hi krishna,
    I guess that you set EMR to use AWS Glue catalog as Hive metadata and Kylin is missing
the AWS lib com.amazonaws.glue in Kylin's classpath. You should find the lib in the EMR cluster
and add it to your class path(maybe $KYLIN_HOME/lib) is OK. 

    If you cannot find the right jar, you may package it manually, repo should be this https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore.
Maybe ask EMR customer service for help should be considered.

==============================================================
Xiaoxiang Yu
Best wishes to you!



> 在 2019年7月16日,10:40,Krishna Bandaru <Krishna.b@scanbuy.com> 写道:
> 
> hi I created Kylin cluster with HA(3 masters and 2 cores)
> <Screen Shot 2019-07-15 at 10.36.18 PM.png>
> java.lang.RuntimeException: java.io.IOException: MetaException(message:Unable to instantiate
a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
not found)
> 	at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:97)
> 	at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:122)
> 	at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:100)
> 	at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131)
> 	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> 	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> 	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> 	at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:111)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: MetaException(message:Unable to instantiate a metastore
client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due
to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
not found)
> 	at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
> 	at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> 	at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:94)
> 	... 10 more
> Caused by: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
not found)
> 	at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClientFactory(HiveUtils.java:525)
> 	at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClient(HiveUtils.java:506)
> 	at org.apache.hive.hcatalog.common.HiveClientCache.getNonCachedHiveMetastoreClient(HiveClientCache.java:99)
> 	at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:318)
> 	at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:315)
> 	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4791)
> 	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3584)
> 	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
> 	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
> 	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
> 	at com.google.common.cache.LocalCache.get(LocalCache.java:3985)
> 	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4788)
> 	at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:315)
> 	at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:277)
> 	at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558)
> 	at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:104)
> 	at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:88)
> 	at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> 	... 12 more
> 
> 
>  I am getting below error please help me to get rid off this.
> even though i set 
> <property>
>   <name>hive.aux.jars.path </name>
>   <value>file:///usr/lib/hive/auxlib/aws-glue-datacatalog-hive2-client.jar</value>
> </property>
> in hive-site.xml i am getting above error while cube designing in step 2(#2 Step Name:
Extract Fact Table Distinct Columns)
> 
> thanks,
> krishna











Mime
View raw message