kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaofeng SHI (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3138) cuboids on-demand build
Date Fri, 29 Dec 2017 05:51:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306009#comment-16306009
] 

Shaofeng SHI commented on KYLIN-3138:
-------------------------------------

Yes we have realized that the static rules are not perfect, especially when there are many
dimensions and the query pattern isn't clear. 

EBay Kylin team has developed a very nice enhancement, which will adjust the cube design based
on query history, this will be introduced in next version
https://issues.apache.org/jira/browse/KYLIN-2727

Regarding this proposal, it need a deep refactor on Kylin core (at this moment, it doesn't
track the status of each cuboid), which might not happen in a near version. Besides, as the
build may take a long time to finish, need evaluate the impact to user experience. 

We're open for such discussion, your comments are always welcomed.

> cuboids on-demand build
> -----------------------
>
>                 Key: KYLIN-3138
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3138
>             Project: Kylin
>          Issue Type: New Feature
>          Components: General, Job Engine, Query Engine, Spark Engine
>    Affects Versions: v2.2.0
>            Reporter: Ruslan Dautkhanov
>            Assignee: Shaofeng SHI
>            Priority: Critical
>
> We just started using Kylin and quite like it so far.
> Although some of the datasets we have a quite like to even consider for OLAP cubing.
> Unless those cuboids will be built on-demand.
> I know some commercial non-open source products do this successfully. 
> This idea is to build a cuboid only when a user actually needs it. 
> So for example, our BI dashboards does a certain rollup, so then a SQL
> query hits Kylin backend. Kylin realizes it hasn't built that particular cuboid just
yet,
> so immediately starts building it. Users has to wait a bit longer first time
> it request that combination of dimensions. But all other requests or requests 
> of other users will be fast from that point on.
> Kylin (or any other OLAP solution) wouldn't be feasible to use on very wide datasets

> unless this on-demand functionality is implemented. For example, some datasets we have
have 100-200 dimensions. And we don't know up front rollups users would want to do.
> Suggesting to have a new dimension build rule "lazy / on-demand". All previous rules
apply. This new rule type would mean, a cuboid for a particular set of dimensions wouldn't
be built up-front if it's marked as "lazy / on-demand". 
> Thoughts / ideas?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message