kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Yang <liy...@apache.org>
Subject Re: Convert Cuboid Data to HFile is taking most of time
Date Fri, 25 May 2018 08:49:58 GMT
Currently there is no Spark version of the step "Convert Cuboid Data to
HFile". The best shot is to tune the Hadoop MR job of converting to HFile.

I would suggest to start by checking parallel-ness of the job. See if there
is enough mappers and reducers started. If not, consider cut the cuboid
into smaller regions, which will increase the number of mappers and
reducers.

On Tue, May 8, 2018 at 2:56 AM, narendracs <narendra.cs@gmail.com> wrote:

> I am using spark engine for cube processing, step to Convert Cuboid Data to
> HFile is taking most of time.
> 14 M input record,  3 dimensions ( 1 UHC) , it took around 28 mins to build
> the cube out of which 20 mins just for Convert Cuboid Data to HFile. I
> noticed this step is running map reduce even though I have selected spark
> as
> engine type.
> is there any way to make this step run on spark instead of MR ?
> also there any configuration which can help to optimize this step?
>
> thanks
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message