kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "朱真龙" <183920...@qq.com>
Subject 回复: Kylin on EMR Installation Error
Date Mon, 25 Dec 2017 03:53:03 GMT
Hi, ShaoFeng,


     thank you  , I'll add the patch to kylin2.1 and retry . thank you again.




------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<shaofengshi@apache.org>;
发送时间: 2017年12月25日(星期一) 中午11:09
收件人: "user"<user@kylin.apache.org>;

主题: Re: Kylin on EMR Installation Error



Hi zhenlong,

Thanks for the detailed analysis. You're correct, there was some inconsistency in Kylin 2.1,
it should have been fixed in 2.2, could you please upgrade Kylin and then verify? 


The related JIRA is:
https://issues.apache.org/jira/browse/KYLIN-2865

https://issues.apache.org/jira/browse/KYLIN-2766



2017-12-25 10:56 GMT+08:00 朱真龙 <183920368@qq.com>:
thank you for your response.after check  https://issues.apache.org/jira/browse/KYLIN-3129
,I find it's not the same problem as mine and  I have found the way to resolved my problem.
I used kylin  between two hadoop cluster ,like this:







its works well when model is not too big. but some model with many row keys does not  .  when
I configured fifteen columns for rowkey and setted dict encode like this :

when running to "Build Dimension Dictionary" step,  I got error as follow( it means  after
writted snapshot to hdfs, when submit these hdfs file to hbase ,the files and hbaseTable are
in different hdfs cluster , one in cluster 1 and one in cluster 2):





through looked into source code the error referred, I found that ,when writting "Dimension
Dictionary" to hbase , kylin will judge if your keyvalue is larger than the configured Max_KeyValue
size (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not setted ,it will
get 10485760(10MB) as default .  if your keyValue size is smaller than hbase.client.keyvalue.maxsize
,then kylin 
will set your key and value into hbase's put object then submit. if your keyvalue is bigger
than hbase.client.keyvalue.maxsize ,it will write snapshot into hdfs first then submit it
via hbase table.  but I think it's maybe a bug on kylin2.1  when use kylin between two hadoop
cluster and build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize. 


when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte snapshot into hdfs(
it got the hdfs path from hive table instead of hbase, and they are in different hdfs cluster)
. so if you are using one hadoop cluster(hive and hbase are on one hadoop cluster ),it will
works well . if you are using two hadoop cluster and your keyvalue size is smaller than hbase.client.keyvalue.maxsize
,it will also works well (use hbase's put object). but if you are using two hadoop cluster
and your keyvalue is bigger than  hbase.client.keyvalue.maxsize , you will get then same error
like mine(beause your hive and hbase are in different hdfs cluster) .
 
        I checked my hbase.client.keyvalue.maxsize in hbase-site.xml, and found ambari set
1MB as default ,and kylin set 10MB as default .so I changed this value to 10MB , and now my
model works well.  If you find a better way to resolve this problem ,please tell me , thank
you. 





apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java











------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<shaofengshi@apache.org>;
发送时间: 2017年12月22日(星期五) 下午5:49
收件人: "user"<user@kylin.apache.org>;

主题: Re: Kylin on EMR Installation Error



Please check https://issues.apache.org/jira/browse/KYLIN-3129

2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org>:
Thanks, I will try EMR 5.8 later; 

At this moment, you can try emr 5.5; one thing to pay attention is, you need copy a configuration
from hbase-site.xml to KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with
yours):


<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
</property>





2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mukund.agilisium@gmail.com>:
Hi

I am using EMR 5.8.0 , the latest one. 


Thanks & Regards
Mukund


On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <shaofengshi@apache.org> wrote:
Hi Mukund, 

What's the EMR version?



Check if there is other version's jar in EMR node, e.g,: 


sudo find / -name  joda-time*.jar




2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mukund.agilisium@gmail.com>:
Hi All
I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR and getting the following
errors.



[root@ip-172-31-35-25 lib]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
[root@ip-172-31-35-25 lib]# ls -al | grep joda
-rw-r--r-- 1 root root  573912 May  6  2016 joda-time-2.2.jar



Looks like the jar is available inside the webinf/lib as well.



Any help is appreciated!!


kylin.log
2017-09-05 12:06:45,097 ERROR [localhost-startStop-1] context.ContextLoader:351 : Context
initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter':
Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]:
Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
......




Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]:
Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1102)
        ... 26 more



   

SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter':
Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]:
Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
 

kylin.out



There is not keystore files in the folder


[root@ip-172-31-35-25 conf]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
[root@ip-172-31-35-25 conf]# ls *.keystore
ls: cannot access *.keystore: No such file or directory





SEVERE: Failed to load keystore type JKS with path conf/.keystore due to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore
(No such file or directory)
java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No
such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
       



Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-7443]]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:838)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)





 






-- 
Best regards,

Shaofeng Shi 史少锋






 
 


 






-- 
Best regards,

Shaofeng Shi 史少锋






 
 






-- 
Best regards,

Shaofeng Shi 史少锋






 








-- 
Best regards,

Shaofeng Shi 史少锋
Mime
View raw message