kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francis Liang <soulvagr...@hotmail.com>
Subject mapreduce resource config does not work at ‘Convert Cuboid Data to HFile’ Step
Date Thu, 13 Sep 2018 07:59:29 GMT
Hi:

I am using kylin-2.4.0-bin-cdh57. For building a segment with more than 100m records, I need
to overwrite some mapreduce configuration to bigger values. I did the following configuration
at Cube level:

kylin.engine.mr.config-override.mapreduce.map.java.opts=-Xmx10g
kylin.engine.mr.config-override.mapreduce.map.memory.mb=10240
kylin.engine.mr.config-override.mapreduce.reduce.java.opts=-Xmx10g
kylin.engine.mr.config-override.mapreduce.reduce.memory.mb=10240

When I was monitoring the resource usage in the cube building process, I found that the above
configuration worked for every step except for the ‘Convert Cuboid Data to HFile’ Step.
In this step, every container on Yarn was still using 1g memory and this caused the cube building
task to fail.

Could anyone help me on this? Thanks ahead!

Best, Feng.

发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用

Mime
View raw message