kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈熹(chenxi07)-技术产品中心 <chenx...@qiyi.com>
Subject RE: How to increase split number for Fact distinct columns when using spark engine?(picture added)
Date Mon, 05 Nov 2018 07:52:12 GMT
Hi, shaofeng:
	Thank you for your reply.
	I've checked the doc, it does not contain tuning for spark engine.

--
Best regards,

Xi Chen

-----Original Message-----
From: ShaoFeng Shi <shaofengshi@apache.org> 
Sent: Monday, November 5, 2018 3:30 PM
To: dev <dev@kylin.apache.org>
Subject: Re: How to increase split number for Fact distinct columns when using spark engine?(picture
added)

Please check this doc:
https://kylin.apache.org/docs/howto/howto_optimize_build.html

陈熹(chenxi07)-技术产品中心 <chenxi07@qiyi.com> 于2018年11月5日周一
下午3:25写道:

> Hi:
>
>        I’m sorry the picture is dead again.
>
>        I upload it as attachment this time
>
>
>
> --
>
> Best regards,
>
>
>
> Xi Chen
>
>
>
>
>
> *From:* 陈熹(chenxi07)-技术产品中心 <chenxi07@qiyi.com>
> *Sent:* Monday, November 5, 2018 3:04 PM
> *To:* dev@kylin.apache.org
> *Subject:* How to increase split number for Fact distinct columns when 
> using spark engine?(picture added)
>
>
>
> Hi, ALL:
>
>        I’m using spark engine to build cube.
>
> Now I found the bottleneck of build time lies in the #3 Step Name: 
> Extract Fact Table Distinct Columns.
>
> When I look into the spark application, I found there is only two 
> splits regardless of how large the input sequence file is.
>
> I wonder how to increase the number of split for this step?
>
> I’m new to spark and any help will be great thanks!
>
>
>
> P.S. Spark job of #3 Step Name: Extract Fact Table Distinct Columns.
>
> --
>
> Best regards,
>
>
>
> Xi Chen
>
>
>
>
>


--
Best regards,

Shaofeng Shi 史少锋
Mime
View raw message