kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: mapreduce resource config does not work at ‘Convert Cuboid Data to HFile’ Step
Date Thu, 13 Sep 2018 10:01:35 GMT
Okay.. they might be different; Just take a try; if the problem still
there, please report a JIRA.

Francis Liang <soulvagrant@hotmail.com> 于2018年9月13日周四 下午4:48写道:

> It says mapreduce.job.queuename config was fixed. Also other
> configurations mentioned in the previous email were also fixed? Thanks!
> Best, Feng.
>
>
>
> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>
>
> ------------------------------
> *发件人:* ShaoFeng Shi <shaofengshi@apache.org>
> *发送时间:* Thursday, September 13, 2018 4:04:31 PM
> *收件人:* user
> *主题:* Re: mapreduce resource config does not work at ‘Convert Cuboid Data
> to HFile’ Step
>
> Hi Feng,
>
> Please check KYLIN-3438. It has been fixed in 2.4.1
>
> Francis Liang <soulvagrant@hotmail.com> 于2018年9月13日周四 下午3:59写道:
>
>> Hi:
>>
>>
>>
>> I am using kylin-2.4.0-bin-cdh57. For building a segment with more than
>> 100m records, I need to overwrite some mapreduce configuration to bigger
>> values. I did the following configuration at Cube level:
>>
>>
>>
>> kylin.engine.mr.config-override.mapreduce.map.java.opts=-Xmx10g
>>
>> kylin.engine.mr.config-override.mapreduce.map.memory.mb=10240
>>
>> kylin.engine.mr.config-override.mapreduce.reduce.java.opts=-Xmx10g
>>
>> kylin.engine.mr.config-override.mapreduce.reduce.memory.mb=10240
>>
>>
>>
>> When I was monitoring the resource usage in the cube building process, I
>> found that the above configuration worked for every step except for the ‘Convert
>> Cuboid Data to HFile’ Step. In this step, every container on Yarn was still
>> using 1g memory and this caused the cube building task to fail.
>>
>>
>>
>> Could anyone help me on this? Thanks ahead!
>>
>>
>>
>> Best, Feng.
>>
>>
>>
>> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>

-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message