kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ma Gang" <mg4w...@163.com>
Subject Re:Re: [Discuss] Lookup table improvement: support global/big lookup table
Date Sun, 01 Apr 2018 09:35:27 GMT
Sure ShaoFeng, I add a storageType field in the SnapshotDesc to support different materialized
storages.
At 2018-04-01 09:57:23, "ShaoFeng Shi" <shaofengshi@apache.org> wrote:
>Thank you Ma Gang; This is a good proposal. Externalizing the lookup
>snapshots will reduce the burden of Kylin query servers.
>
>My only comment is, does this implementation support extension? You know
>since Kylin 1.5, Kylin has the plug-in architecture, HBase is one
>implementation for the storage. Kylin core modules don't directly depend on
>HBase anymore;  so please take this into consideration when you implement
>it.
>
>2018-03-30 17:57 GMT+08:00 magang <mg4work@163.com>:
>
>> Hi all,
>>
>> There are two limitations for current lookup table design:
>>
>> 1. lookup table size is limited, because table snapshot need to be cached
>> in
>> Kylin server, too large snapshot table will break Kylin server, also the
>> snapshot building may take very long time when it is too large.
>> 2. each segment has its own lookup table snapshot, but some users may need
>> a
>> global snapshot table, which means when the global table is updated, the
>> query for all segments need to reflect the change.
>>
>> To resolve the above limitations, I have created ticket:
>> https://issues.apache.org/jira/browse/KYLIN-3221
>> <https://issues.apache.org/jira/browse/KYLIN-3221>  , and put initial new
>> design doc there, any comments and suggestions are welcome.
>>
>> --
>> Sent from: http://apache-kylin.74782.x6.nabble.com/
>>
>
>
>
>-- 
>Best regards,
>
>Shaofeng Shi 史少锋
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message