kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaoxiang Yu <xiaoxiang...@kyligence.io>
Subject Re: Error on EMR
Date Tue, 03 Dec 2019 14:41:18 GMT
Hi, 
   I have successfully deployed latest version of Kylin(3.0.beta) on AWS EMR 5.27 and build
a few cubes successfully, maybe you can have a try? 
   The cluster is created by CLI looks like this, and I deployed Kylin on MASTER node:

aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Pig Name=Spark Name=Sqoop
Name=Tez Name=Zeppelin Name=ZooKeeper Name=Ganglia\
	--release-label emr-5.27.0 \
	--instance-groups '[{"InstanceCount":4,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":200,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"CORE","InstanceType":"m4.2xlarge","Name":"Worker
Cluster"},{"InstanceCount":1,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":100,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"MASTER","InstanceType":"c4.4xlarge","Name":"MasterQuery"}]'
\
	--configurations '[{"Classification":"hdfs-site","Properties":{"dfs.replication":"2"}}]'
\
	--ebs-root-volume-size 100 \	--enable-debugging \
	--name 'BenchmarkCluster' \
	--scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
	--region cn-northwest-1

----------------
Best wishes,
Xiaoxiang Yu 
 

在 2019/12/2 20:38,“Tanmay Movva”<tanmay.movva@razorpay.com.INVALID> 写入:

    Hello,
    
    We have installed kylin on our EMR master along with hbase, hadoop and
    hive. Using download-spark.sh from KYLIN_HOME/bin I have installed spark.
    As mentioned in "Install KYLIN on AWS EMR" guide we have followed the steps
    to configure Kylin working dir and hbase storage as S3 and also made the
    necessary zkquorum changes.
    
    When we run the sample.sh or check-env.sh we don't get any errors. But when
    we run the cube build job from UI, the job fails at stage-2 "Redistribute
    Flat Hive Tables". As the job "Create Intermediate Hive tables" has been
    completed successfully I don't think there has been any error with Hive.
    
    Can anyone help us with this? Thank You.
    
    
    java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
    	at org.apache.kylin.source.hive.CLIHiveClient.<init>(CLIHiveClient.java:47)
    	at org.apache.kylin.source.hive.HiveClientFactory.getHiveClient(HiveClientFactory.java:27)
    	at org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.computeRowCount(RedistributeFlatHiveTableStep.java:40)
    	at org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.doWork(RedistributeFlatHiveTableStep.java:91)
    	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
    	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
    	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
    	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.ClassNotFoundException:
    org.apache.hadoop.hive.conf.HiveConf
    	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
    	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
    	... 11 more
    
    -- 
    Regards,
    Tanmay Krishna Movva
    Razorpay
    

Mime
View raw message