kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billy Liu <billy...@apache.org>
Subject Re: Re: [Discuss] Lookup table improvement: support global/big lookup table
Date Sun, 01 Apr 2018 15:12:37 GMT
Hi Magang,

Very good proposal. I have one few questions here. In item 2, you
mentioned global level snapshot and segment level snapshot. This is
similar scenario with handling slowly change dimensions. Is there a
configuration item on Cube to tell which snapshot would be used? Could
the segment level snapshot be refreshed independently? Could we show
the snapshot size, build time, storage path on the GUI?

With Warm regards

Billy Liu


2018-04-01 18:19 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org>:
> Thank you, Gang! Looking forward to seeing this feature in Kylin.
>
> 2018-04-01 17:35 GMT+08:00 Ma Gang <mg4work@163.com>:
>
>> Sure ShaoFeng, I add a storageType field in the SnapshotDesc to support
>> different materialized storages.
>> At 2018-04-01 09:57:23, "ShaoFeng Shi" <shaofengshi@apache.org> wrote:
>> >Thank you Ma Gang; This is a good proposal. Externalizing the lookup
>> >snapshots will reduce the burden of Kylin query servers.
>> >
>> >My only comment is, does this implementation support extension? You know
>> >since Kylin 1.5, Kylin has the plug-in architecture, HBase is one
>> >implementation for the storage. Kylin core modules don't directly depend
>> on
>> >HBase anymore;  so please take this into consideration when you implement
>> >it.
>> >
>> >2018-03-30 17:57 GMT+08:00 magang <mg4work@163.com>:
>> >
>> >> Hi all,
>> >>
>> >> There are two limitations for current lookup table design:
>> >>
>> >> 1. lookup table size is limited, because table snapshot need to be
>> cached
>> >> in
>> >> Kylin server, too large snapshot table will break Kylin server, also the
>> >> snapshot building may take very long time when it is too large.
>> >> 2. each segment has its own lookup table snapshot, but some users may
>> need
>> >> a
>> >> global snapshot table, which means when the global table is updated, the
>> >> query for all segments need to reflect the change.
>> >>
>> >> To resolve the above limitations, I have created ticket:
>> >> https://issues.apache.org/jira/browse/KYLIN-3221
>> >> <https://issues.apache.org/jira/browse/KYLIN-3221>  , and put initial
>> new
>> >> design doc there, any comments and suggestions are welcome.
>> >>
>> >> --
>> >> Sent from: http://apache-kylin.74782.x6.nabble.com/
>> >>
>> >
>> >
>> >
>> >--
>> >Best regards,
>> >
>> >Shaofeng Shi 史少锋
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋

Mime
View raw message