kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Tardío Olmos <roberto.tar...@stratebi.com>
Subject Re: A problem in cube building time
Date Thu, 27 Apr 2017 11:08:00 GMT
Hi Zhuoran,

I faced a similar problem about cube building time. I think that depends 
on the cardinality of the 2 dimensions you add. If some of these has a 
big cardinality (eg. in my use case about 500.000 rows, Customer 
Dimension) the number of combinations Kylin need to build the cube 
increases a lot.

Some things you could try to reduce cube building time and size:

  * Define all Dimension tables attributes as a Derived Dimensions. In
    this cases you can not use Hierarchy optimization in Agg Group. The
    query latency in queries that use derived attributes will be less
    optimal than using Agg Group Hierarchies (with Normal Dimensions),
    but in some cases the differences in query latency are acceptable
    (in my case between 2 and 6 seconds more, depending of the query).
    Cube size and building time will be reduced.
  * Use "Shard By" in Rowkey for High Cardinality Dimensions. I have not
    been able to test it yet, but as indicated at
    https://kylin.apache.org/docs16/howto/howto_optimize_build.html
    should work fine. This helps to reduce cube building time.

I hope to help you, I'm also learning to use Kylin.

Kind Regards,

El 27/04/2017 a las 12:46, 吕卓然 escribió:
>
> Hi all,
>
> Currently I am using Kylin 1.6.1 and I face a problem about cube 
> building time. I have one fact table and two lookup tables. When I set 
> 13 normal dimensions and 15 derived dimensions and two measures (count 
> and count distinct). The step3 in building takes around 20mins and the 
> entire building takes around 1 hour. This is good.
>
> However, when I try to increase to 15 normal dimensions and 15 derived 
> dimensions and two measures(count and count distinct). The step3 in 
> building takes around 240mins and the entire building takes forever….
>
> BTW, I have a hierarchy dimension which has 4 normal dimensions.
>
> I am really confusing about this.  Does 13 normal dimensions become a 
> bottleneck in building cube?
>
> Thanks a lot!
>
> Zhuoran
>

-- 
*Roberto Tardío Olmos*
/Senior Big Data & Business Intelligence Consultant/

Avenida de Brasil, 17, Planta 16.

28020 Madrid

Fijo: 91.788.34.10


Mime
View raw message