kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ma Gang" <>
Subject Re:Re: [Discuss] Lookup table improvement: support global/big lookup table
Date Sun, 01 Apr 2018 09:35:27 GMT
Sure ShaoFeng, I add a storageType field in the SnapshotDesc to support different materialized
At 2018-04-01 09:57:23, "ShaoFeng Shi" <> wrote:
>Thank you Ma Gang; This is a good proposal. Externalizing the lookup
>snapshots will reduce the burden of Kylin query servers.
>My only comment is, does this implementation support extension? You know
>since Kylin 1.5, Kylin has the plug-in architecture, HBase is one
>implementation for the storage. Kylin core modules don't directly depend on
>HBase anymore;  so please take this into consideration when you implement
>2018-03-30 17:57 GMT+08:00 magang <>:
>> Hi all,
>> There are two limitations for current lookup table design:
>> 1. lookup table size is limited, because table snapshot need to be cached
>> in
>> Kylin server, too large snapshot table will break Kylin server, also the
>> snapshot building may take very long time when it is too large.
>> 2. each segment has its own lookup table snapshot, but some users may need
>> a
>> global snapshot table, which means when the global table is updated, the
>> query for all segments need to reflect the change.
>> To resolve the above limitations, I have created ticket:
>> <>  , and put initial new
>> design doc there, any comments and suggestions are welcome.
>> --
>> Sent from:
>Best regards,
>Shaofeng Shi 史少锋
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message