kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Kylin on EMR Installation Error
Date Mon, 25 Dec 2017 05:59:53 GMT
Hi zhenlong,

I'm afraid that patch is not all the changes; suggest you download the
formal 2.2 release.

2017-12-25 11:53 GMT+08:00 朱真龙 <183920368@qq.com>:

> Hi, ShaoFeng,
>
>      thank you  , I'll add the patch to kylin2.1 and retry . thank you
> again.
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<shaofengshi@apache.org>;
> *发送时间:* 2017年12月25日(星期一) 中午11:09
> *收件人:* "user"<user@kylin.apache.org>;
> *主题:* Re: Kylin on EMR Installation Error
>
> Hi zhenlong,
>
> Thanks for the detailed analysis. You're correct, there was some
> inconsistency in Kylin 2.1, it should have been fixed in 2.2, could you
> please upgrade Kylin and then verify?
>
> The related JIRA is:
> https://issues.apache.org/jira/browse/KYLIN-2865
> https://issues.apache.org/jira/browse/KYLIN-2766
>
> 2017-12-25 10:56 GMT+08:00 朱真龙 <183920368@qq.com>:
>
>> thank you for your response.after check  https://issues.apache.o
>> rg/jira/browse/KYLIN-3129 ,I find it's not the same problem as mine and
>> I have found the way to resolved my problem. I used kylin  between two
>> hadoop cluster ,like this:
>>
>>
>>
>> its works well when model is not too big. but some model with many row
>> keys does not  .  when I configured fifteen columns for rowkey and setted
>> dict encode like this :
>> when running to "Build Dimension Dictionary" step,  I got error as
>> follow( it means  after writted snapshot to hdfs, when submit these hdfs
>> file to hbase ,the files and hbaseTable are in different hdfs cluster , one
>> in cluster 1 and one in cluster 2):
>>
>>
>> through looked into source code the error referred, I found that ,when
>> writting "Dimension Dictionary" to hbase , kylin will judge if your
>> keyvalue is larger than the configured Max_KeyValue size
>> (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not
>> setted ,it will get 10485760(10MB) as default .  if your keyValue size is
>> smaller than hbase.client.keyvalue.maxsize ,then kylin
>> will set your key and value into hbase's put object then submit. if your
>> keyvalue is bigger than hbase.client.keyvalue.maxsize ,it will write
>> snapshot into hdfs first then submit it via hbase table.  but I think it's
>> maybe a bug on kylin2.1  when use kylin between two hadoop cluster and
>> build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize.
>>
>> when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte
>> snapshot into hdfs( it got the hdfs path from hive table instead of hbase,
>> and they are in different hdfs cluster) . so if you are using one hadoop
>> cluster(hive and hbase are on one hadoop cluster ),it will works well . if
>> you are using two hadoop cluster and your keyvalue size is smaller than
>> hbase.client.keyvalue.maxsize ,it will also works well (use hbase's put
>> object). but if you are using two hadoop cluster and your keyvalue is
>> bigger than  hbase.client.keyvalue.maxsize , you will get then same error
>> like mine(beause your hive and hbase are in different hdfs cluster) .
>>
>>         I checked my hbase.client.keyvalue.maxsize in hbase-site.xml, and
>> found ambari set 1MB as default ,and kylin set 10MB as default .so I
>> changed this value to 10MB , and now my model works well.  If you find a
>> better way to resolve this problem ,please tell me , thank you.
>>
>>
>> apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\
>> main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "ShaoFeng Shi";<shaofengshi@apache.org>;
>> *发送时间:* 2017年12月22日(星期五) 下午5:49
>> *收件人:* "user"<user@kylin.apache.org>;
>> *主题:* Re: Kylin on EMR Installation Error
>>
>> Please check https://issues.apache.org/jira/browse/KYLIN-3129
>>
>> 2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org>:
>>
>>> Thanks, I will try EMR 5.8 later;
>>>
>>> At this moment, you can try emr 5.5; one thing to pay attention is, you
>>> need copy a configuration from hbase-site.xml to
>>> KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with
>>> yours):
>>>
>>> <property>
>>>   <name>hbase.zookeeper.quorum</name>
>>>   <value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
>>> </property>
>>>
>>>
>>> 2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mukund.agilisium@gmail.com>
>>> :
>>>
>>>> Hi
>>>>
>>>> I am using EMR 5.8.0 , the latest one.
>>>>
>>>> Thanks & Regards
>>>> Mukund
>>>>
>>>> On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <shaofengshi@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Mukund,
>>>>>
>>>>> What's the EMR version?
>>>>>
>>>>> Check if there is other version's jar in EMR node, e.g,:
>>>>>
>>>>> sudo find / -name  joda-time*.jar
>>>>>
>>>>>
>>>>> 2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mukund.agilisium@gmail.com
>>>>> >:
>>>>>
>>>>>> Hi All
>>>>>>
>>>>>> I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR
>>>>>> and getting the following errors.
>>>>>>
>>>>>> [root@ip-172-31-35-25 lib]# pwd
>>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
>>>>>> [root@ip-172-31-35-25 lib]# ls -al | grep joda
>>>>>> -rw-r--r-- 1 root root  573912 May  6  2016 joda-time-2.2.jar
>>>>>>
>>>>>> Looks like the jar is available inside the webinf/lib as well.
>>>>>>
>>>>>> Any help is appreciated!!
>>>>>>
>>>>>> *kylin.log*
>>>>>> 2017-09-05 12:06:45,097 ERROR [localhost-startStop-1]
>>>>>> context.ContextLoader:351 : Context initialization failed
>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter':
>>>>>> Instantiation of bean failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>> instantiate [org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>> a/time/format/DateTimeFormatter;
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>>> ory.java:306)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>>> tSingletonBeanRegistry.java:230)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>>> y.java:302)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>>>         at org.springframework.context.su
>>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>>> tion(AbstractApplicationContext.java:839)
>>>>>>         at org.springframework.context.su
>>>>>> pport.AbstractApplicationContext.refresh(AbstractApplication
>>>>>> Context.java:538)
>>>>>>         at org.springframework.web.contex
>>>>>> t.ContextLoader.configureAndRefreshWebApplicationContext(Con
>>>>>> textLoader.java:444)
>>>>>>         at org.springframework.web.contex
>>>>>> t.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
>>>>>> ......
>>>>>>
>>>>>>
>>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>>> Failed to instantiate [org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>> a/time/format/DateTimeFormatter;
>>>>>>         at org.springframework.beans.Bean
>>>>>> Utils.instantiateClass(BeanUtils.java:163)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.SimpleInstantiationStrategy.instantiate(SimpleIn
>>>>>> stantiationStrategy.java:89)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>> an(AbstractAutowireCapableBeanFactory.java:1102)
>>>>>>         ... 26 more
>>>>>>
>>>>>>
>>>>>> SEVERE: Exception sending context initialized event to listener
>>>>>> instance of class org.springframework.web.contex
>>>>>> t.ContextLoaderListener
>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter':
>>>>>> Instantiation of bean failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>> instantiate [org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>> a/time/format/DateTimeFormatter;
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>>> ory.java:306)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>>> tSingletonBeanRegistry.java:230)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>>> y.java:302)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>         at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>>>         at org.springframework.context.su
>>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>>> tion(AbstractApplicationContext.java:839)
>>>>>>
>>>>>> *kylin.out*
>>>>>>
>>>>>> There is not keystore files in the folder
>>>>>>
>>>>>> [root@ip-172-31-35-25 conf]# pwd
>>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
>>>>>> [root@ip-172-31-35-25 conf]# ls *.keystore
>>>>>> ls: cannot access *.keystore: No such file or directory
>>>>>>
>>>>>>
>>>>>> SEVERE: Failed to load keystore type JKS with path conf/.keystore
due
>>>>>> to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No
>>>>>> such file or directory)
>>>>>> java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore
>>>>>> (No such file or directory)
>>>>>>         at java.io.FileInputStream.open0(Native Method)
>>>>>>         at java.io.FileInputStream.open(FileInputStream.java:195)
>>>>>>         at java.io.FileInputStream.<init>(FileInputStream.java:138)
>>>>>>         at java.io.FileInputStream.<init>(FileInputStream.java:93)
>>>>>>
>>>>>>
>>>>>> Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService
>>>>>> initInternal
>>>>>> SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
>>>>>> org.apache.catalina.LifecycleException: Failed to initialize
>>>>>> component [Connector[HTTP/1.1-7443]]
>>>>>>         at org.apache.catalina.util.Lifec
>>>>>> ycleBase.init(LifecycleBase.java:106)
>>>>>>         at org.apache.catalina.core.Stand
>>>>>> ardService.initInternal(StandardService.java:560)
>>>>>>         at org.apache.catalina.util.Lifec
>>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>>>         at org.apache.catalina.core.Stand
>>>>>> ardServer.initInternal(StandardServer.java:838)
>>>>>>         at org.apache.catalina.util.Lifec
>>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi 史少锋
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message