kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: OutOfMemoryError happened when build cube
Date Mon, 28 Aug 2017 13:11:55 GMT
Step 2 is a hive operation, you may need check Hive configuration to
allocate more memory to containers. You can also customize
conf/kylin_hive_conf.xml, as it will be applied before running Hive
command.

Step 4 is in Kylin process; If this step got OOM, you need check whether
the cube is using "dict" encoding for a high cardinality column. As we
know, "dict" doesn't fit for high cardinality, you need revisit the cube
design, and change to "fixed_length" or "integer" encoding for that column.

Also, check this doc, it may help:
https://kylin.apache.org/docs21/howto/howto_optimize_build.html


2017-08-28 13:54 GMT+08:00 周湘伦 <jayzhouxl@gmail.com>:

> Sorry,i got these problems in the step 2:Redistribute Flat Hive
> Table,sometimes in the step 4:Build Dimension Dictionary,it is
> uncertain.But the step 2  occurs a little more.
>
> 2017-08-28 11:19 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org>:
>
> > In which step you got this error? This is also important for analysis the
> > issue.
> >
> > 2017-08-28 10:01 GMT+08:00 周湘伦 <jayzhouxl@gmail.com>:
> >
> > > Hi,all
> > > When i built cube,some error hapeend(java.lang.OutOfMemoryError).
> > >
> > > The version as belows:
> > > jdk1.8.0_131,kylin-2.0.0
> > >
> > > The ram is 8g.
> > >
> > > Some configure properties belows:
> > > <property>
> > >         <name>mapreduce.map.memory.mb</name>
> > >         <value>5120</value>
> > >         <description></description>
> > >     </property>
> > >
> > >     <property>
> > >         <name>mapreduce.map.java.opts</name>
> > >         <value>-Xmx4096m -XX:OnOutOfMemoryError='kill -9 %p'</value>
> > >         <description></description>
> > >     </property>
> > >
> > > The log is :
> > > # java.lang.OutOfMemoryError: Java heap space
> > > # -XX:OnOutOfMemoryError="kill -9 %p"
> > > #   Executing /bin/sh -c "kill -9 26605"...
> > >
> > > Then i check the log of gc:
> > > ...
> > > 2017-08-27T11:16:41.112+0800: 85553.167: [Full GC (Allocation Failure)
> > > 2017-08-27T11:16:41.112+0800: 85553.168: [CMS2017-08-27T11:16:43.369+
> > 0800:
> > > 85555.424: [CMS-concurrent-mark: 2.287/2.288 secs] [Times: user=2.34
> > > sys=0.00, real=2.29 secs]
> > >  (concurrent mode failure): 3853567K->3853567K(3853568K), 7.3121409
> secs]
> > > 4080766K->4063993K(4080768K), [Metaspace: 109873K->109873K(1148928K)],
> > > 7.3122661 secs] [Times: user=7.31 sys=0.00, real=7.32 secs]
> > > 2017-08-27T11:16:48.453+0800: 85560.508: [Full GC (Allocation Failure)
> > > 2017-08-27T11:16:48.453+0800: 85560.508: [CMS:
> > > 3853567K->3853567K(3853568K), 5.1092943 secs]
> > 4080766K->4064637K(4080768K),
> > > [Metaspace: 109873K->109873K(1148928K)], 5.1094013 secs] [Times:
> > user=5.09
> > > sys=0.00, real=5.11 secs]
> > > 2017-08-27T11:16:53.563+0800: 85565.618: [GC (CMS Initial Mark) [1
> > > CMS-initial-mark: 3853567K(3853568K)] 4064639K(4080768K), 0.0862505
> secs]
> > > [Times: user=0.16 sys=0.00, real=0.08 secs]
> > > 2017-08-27T11:16:53.649+0800: 85565.704: [CMS-concurrent-mark-start]
> > > 2017-08-27T11:16:53.678+0800: 85565.734: [Full GC (Allocation Failure)
> > > 2017-08-27T11:16:53.679+0800: 85565.734: [CMS2017-08-27T11:16:55.983+
> > 0800:
> > > 85568.038: [CMS-concurrent-mark: 2.333/2.334 secs] [Times: user=2.37
> > > sys=0.01, real=2.34 secs]
> > > ...
> > >
> > > Maybe is the error of java Garbage Collection,so i how to solve the
> > > problem?(maybe modify the properties)
> > >
> > > Thanks a lot.
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message