kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoxiang Yu" <x...@apache.org>
Subject Re:Kylin Start fails with NoClassDefFoundError on AWS EMR Edge Node
Date Thu, 09 Apr 2020 10:24:25 GMT
Hi friend, 
    I have a suggestion and I am not sure if it is the best practice, and I don't know if
it works. But maybe you can have a try if you are willing to do it.
    I guess maybe you have installed Kylin successfully on Master node of EMR cluster, right?
After kylin instance was started successfully(if you are using the kylin 3.0 or above), you
will see some files with cached-*-dependency.sh under $KYLIN_HOME/bin, in all these files,
you can find the location of jars that is needed by Kylin. You may try copy the jars from
master node to edge node in the same folder, then restart Kylin in edge node.
    On the another hand, you don't need to modify the property "kylin.job.mr.lib.dir" when
starting a Kylin instance.


    Following :


  cached-hadoop-conf-dir.sh
  cached-hbase-dependency.sh
  cached-hive-dependency.sh
  cached-kafka-dependency.sh
  cached-spark-dependency.sh


   [root@cdh-master all]# cat bin/cached-hive-dependency.sh
export hive_dependency=/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/conf:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-jdbc-standalone.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/junit-4.11.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/metrics-jvm-3.0.2.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-cli-1.1.0-cdh5.7.6.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib
 .....
....






















--

Best wishes to you ! 
From ´╝ÜXiaoxiang Yu





At 2020-04-09 09:22:59, "mvishnubhatta" <mvishnubhatta@argusinformation.com> wrote:
>Hi,
>
>I am trying to set up a Kylin server on an AWS EMR edge node. When doing
>that the web URL throws the error that "The origin server did not find a
>current representation for the target resource or is not willing to disclose
>that one exists".
>
>The tomcat localhost log file shows a NoClassDefFoundError for the class
>org/apache/hadoop/hive/metastore/api/NoSuchObjectException 
>
>But I see this class in the hive library
>/usr/lib/hive/lib/hive-metastore-2.3.5-amzn-0.jar and the path is exported
>as HIVE_LIB and as HIVE_HOME (used HIVE_LIB based on suggestion here
>https://issues.apache.org/jira/browse/KYLIN-2511).
>
>This appears to be a case where some CLASSPATH somewhere is not correctly
>including this path.
>
>So, I tried modifying the kylin.properties file to add a property called
>kylin.job.mr.lib.dir pointing to the /usr/lib/hive/lib/ but that didnt help
>either. (This too was based on another suggested workaround here).
>
>Can someone point out what I am missing?
>
>Here are the full details:
>
>Hadoop version is Hadoop 2.8.5-amzn-4
>Kylin version is 3.0.1
>Hive 2.3.5-amzn-0
>
>I have an AWS EMR Cluster and a separate EC2 machine that is set up as an
>edge node. From this edge node I am able to run hive queries to connect to
>the EMR Master node. Hbase also works on this node.
>
>On this edge node, I are trying to start a Kylin server. I followed the
>instructions given here: 
>https://kylin.apache.org/docs/install/kylin_aws_emr.html
><https://kylin.apache.org/docs/install/kylin_aws_emr.html>  
>
>I exported the following variables:
>export KYLIN_HOME=/usr/local/kylin/apache-kylin-3.0.1-bin-hbase1x
>#export HADOOP_HOME=/usr/lib/hadoop/etc/hadoop
>export HADOOP_HOME=/usr/lib/hadoop
>export HBASE_HOME=/usr/lib/hbase/
>export HIVE_HOME=/usr/lib/hive/lib/
>export HADOOP_CONF_DIR=/etc/hadoop/conf
>export HIVE_LIB=/usr/lib/hive/lib
>export HIVE_CONF=/etc/hive/conf
>export HCAT_HOME=/usr/lib/hive-hcatalog
>
>
>The command I run is: $KYLIN_HOME/bin/kylin.sh start
>
>After starting it, I get this message:
>
>/A new Kylin instance is started by hadoop. To stop it, run 'kylin.sh stop'
>Check the log at
>/usr/local/kylin/apache-kylin-3.0.1-bin-hbase1x/logs/kylin.log
>Web UI is at http://ip-xxxxxxxxxxxxxxxxxxxxx.compute.internal:7070/kylin/
>
>When I navigate to the URL, I get the following error.
>
>"The origin server did not find a current representation for the target
>resource or is not willing to disclose that one exists."
>
>When I look into the log file which is at
>$KYLIN_HOME/tomcat/logs/localhost.2020-04-08.log I see the following error:
>
>SEVERE: Exception sending context initialized event to listener instance of
>class org.springframework.web.context.ContextLoaderListener
>org.springframework.beans.factory.BeanCreationException: Error creating bean
>with name
>'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping':
>Invocation of init method failed; nested exception is
>java.lang.NoClassDefFoundError:
>org/apache/hadoop/hive/metastore/api/NoSuchObjectException
>        at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
>        at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
>        at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>        at
>org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>        at
>org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>        at
>org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>        at
>org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>        at
>org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
>        at
>org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
>        at
>org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
>        at
>org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
>        at
>org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
>        at
>org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
>        at
>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
>        at
>org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
>        at
>org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
>        at
>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1018)
>        at
>org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:994)
>        at
>org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
>        at
>org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
>        at
>org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2021)
>        at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>        at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at java.lang.Thread.run(Thread.java:748)
>Caused by: java.lang.NoClassDefFoundError:
>org/apache/hadoop/hive/metastore/api/NoSuchObjectException
>        at java.lang.Class.getDeclaredMethods0(Native Method)
>        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
>        at java.lang.Class.getDeclaredMethods(Class.java:1975)
>        at
>org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613)
>        at
>org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
>        at
>org.springframework.core.MethodIntrospector.selectMethods(MethodIntrospector.java:68)
>        at
>org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.detectHandlerMethods(AbstractHandlerMethodMapping.java:230)
>        at
>org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.initHandlerMethods(AbstractHandlerMethodMapping.java:214)
>        at
>org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.afterPropertiesSet(AbstractHandlerMethodMapping.java:184)
>        at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.afterPropertiesSet(RequestMappingHandlerMapping.java:127)
>        at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
>        at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
>        ... 25 more
>Caused by: java.lang.ClassNotFoundException:
>org.apache.hadoop.hive.metastore.api.NoSuchObjectException
>        at
>org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
>
>
>
>
>--
>Sent from: http://apache-kylin.74782.x6.nabble.com/
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message