kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Configuration Overwrites the container memory for MR task not work
Date Mon, 30 Jul 2018 07:35:16 GMT
Hi Xixin,

The error was occurred in Hive, while the "kylin.engine.mr.config-override"
doesn't change Hive's behavior, it only affects the MR jobs that directly
submitted by Kylin (not in-directly by Hive).

To make it effective, you can try to add the mr configurations to
"conf/kylin_hive_conf.xml", or override with prefix
"kylin.source.hive.config-override" in the Cube's advanced page.

2018-07-19 13:47 GMT+08:00 赵希欣 <zxxbuaa@163.com>:

>
> Hi Kylin users/developers
> Kylin 2.3.0, hadoop 2.6.5 cdh 5.12
> A error occurred when I built a cube at step 2 Redistribute Flat Hive
> Table. I checked the MR log and the log shows that reducers containers is
> running beyond physical memory limits
> below is detail logs:
> Diagnostic Messages for this Task:
>
> *Container
> [pid=50920,containerID=container_1518617687620_384770_01_000287] is running
> beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical
> memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container. *
> then I try to overwrites the related config
> kylin.engine.mr.config-override.mapreduce.reduce.memory.mb = 4096
> kylin.engine.mr.config-override.mapreduce.reduce.java.opts = -Xmx3g
>
> then built the cube again, the the same error came again.
> I check the yarn config mapreduce.reduce.memory.mb = 2G. this time I'm
> fully confused. where kylin get the 1GB physical memory config limit? why
> the overwrites not work?
>
>
> --
> BR
> Xixin
>
>
>
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message