kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lk_hadoop"<lk_had...@163.com>
Subject 回复: Re: 答复: can't pass step Build Cube In-Mem
Date Thu, 11 Apr 2019 10:00:12 GMT
thank you ~ @Long Chao 

2019-04-11 

lk_hadoop 



发件人:Long Chao <chao.long0101@gmail.com>
发送时间:2019-04-11 17:56
主题:Re: 答复: can't pass step Build Cube In-Mem
收件人:"dev"<dev@kylin.apache.org>
抄送:

Hi lk, 
     Optimize job will only build the newly generated cuboids in the 
recommended cuboid list, usually the amount of them is not too large. 
     So, by default, we use inmem algorithm to build those new cuboids, but 
now the algorithm can't be overwritten by properties file. 

     And I create a jira for this problem to make the algorithm 
configurable. https://issues.apache.org/jira/browse/KYLIN-3950 

On Thu, Apr 11, 2019 at 5:49 PM lk_hadoop <lk_hadoop@163.com> wrote: 

> I think that's not too much : 
> 
> Cuboid Distribution 
> Current Cuboid Distribution 
> [Cuboid Count: 49] [Row Count: 1117994636] 
> 
> Recommend Cuboid Distribution 
> [Cuboid Count: 168] [Row Count: 464893216] 
> 
> 
> 2019-04-11 
> 
> lk_hadoop 
> 
> 
> 
> 发件人:Na Zhai <na.zhai@kyligence.io> 
> 发送时间:2019-04-11 17:42 
> 主题:答复: can't pass step Build Cube In-Mem 
> 收件人:"dev@kylin.apache.org"<dev@kylin.apache.org> 
> 抄送: 
> 
> Hi, lk_hadoop. 
> 
> 
> 
> Does Cube planner recommend too many cuboid? If so, it may cause OOM. 
> 
> 
> 
> 
> 
> 发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用

> 
> 
> 
> ________________________________ 
> 发件人: lk_hadoop <lk_hadoop@163.com> 
> 发送时间: Tuesday, April 9, 2019 9:21:59 AM 
> 收件人: dev 
> 主题: can't pass step Build Cube In-Mem 
> 
> hi,all : 
>    I'm using kylin-2.6.1-cdh57, and the source row count is 500 million,I 
> can success build cube . 
>    but when I use the cube planner , it has one step : Build Cube In-Mem 
> for job :OPTIMIZE CUBE 
>    the config about the kylin_job_conf_inmem.xml is : 
> 
>    <property> 
>         <name>mapreduce.map.memory.mb</name> 
>         <value>9216</value> 
>         <description></description> 
>     </property> 
> 
>     <property> 
>         <name>mapreduce.map.java.opts</name> 
>         <value>-Xmx8192m -XX:OnOutOfMemoryError='kill -9 %p'</value> 
>         <description></description> 
>     </property> 
> 
>     <property> 
>         <name>mapreduce.job.is-mem-hungry</name> 
>         <value>true</value> 
>     </property> 
> 
>     <property> 
>         <name>mapreduce.job.split.metainfo.maxsize</name> 
>         <value>-1</value> 
>         <description>The maximum permissible size of the split metainfo 
> file. 
>             The JobTracker won't attempt to read split metainfo files 
> bigger than 
>             the configured value. No limits if set to -1. 
>         </description> 
>     </property> 
> 
>     <property> 
>         <name>mapreduce.job.max.split.locations</name> 
>         <value>2000</value> 
>         <description>No description</description> 
>     </property> 
> 
>     <property> 
>         <name>mapreduce.task.io.sort.mb</name> 
>         <value>200</value> 
>         <description></description> 
>     </property> 
> 
> 
>     finally the map job will be killed for OnOutOfMemoryError  , but when 
> I giev more mem for map job , I will get another error : 
> java.nio.BufferOverflowException 
> 
>     why kylin will run the job inmem ? how can I avoid it ? 
> 
> 
> 
> 2019-04-08 
> 
> 
> lk_hadoop 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message