kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billy Liu <billy...@apache.org>
Subject Re: 事实表历史数据的更新,如何更新cube
Date Tue, 03 Sep 2019 08:51:28 GMT
数据回填(write-back)目前是不支持的,也不在计划上。

做到更高效的历史数据更新,就是要尽量不重刷无变化的数据,而变化的数据总要重新刷新。Kylin刷新的单位是Segment。如果不需要“支付”“开票”“创建”之间的关联计算,可以把这些字段拆在不同的Cube中,分别做Partition字段,刷新的时候要重刷新旧两个时间段。但这个可能的潜在风险是不同Cube之间,数据的不一致。

一般来说,数据的更新具备冷热属性,即热数据经常更新,冷数据不更新,因此可以针对冷热,设计不同规模的Segment,减少刷新时的复杂度。

With Warm regards

Billy Liu


Yun, Henry (BJ/DEL) <henry.yun@kpmg.com> 于2019年9月3日周二 下午12:54写道:

> 这个问题非常具有普遍意义。我司在开发类似财务指标平台的时候也遇到类似问题。
>
>
>
> 我们不仅需要从Cube里面查询多维度财务指标数据(例如:公司A-华北区-产品A-已支付订单数-10000
> 单),有些没有前端系统的汇报单位还要实时填报多维数据(例如:公司A-西南区-产品A-已支付订单数-
> ???单)。西南区的负责人填报后需要能够实时汇总到更上一层的区域当中,以供合并查询。
>
>
>
> 过去我们在Hyperion Essbase中是可以在多维数据库中进行填报的。现在要转型到Kylin,不知填报问题如何解决?请各位大师指点。
>
>
>
> 谢谢!
>
> 运宁
>
>
>
> *From:* 王刚 [mailto:mail@wanggang1987.com]
> *Sent:* Tuesday, September, 03, 2019 11:53 AM
> *To:* dev@kylin.apache.org
> *Cc:* user@kylin.apache.org
> *Subject:* Re: 事实表历史数据的更新,如何更新cube
>
>
>
> 补充一下这个场景:
>
>
>
> 有事实表orders,以创建时间作为分区,有已支付订单数,已开票订单数两个指标,都以订单创建时间作为统计分区。
>
>
>
>
>
> order1这条数据,20190801和20190901分别对时间分区为20190701的数据支付时间和开票时间做了更新。
>
>
>
> 那么20190701这个时间分区的已支付订单数和已开票订单数两个指标都需要更新。
>
>
>
> 在这个场景中,我们如何进行cube的更新呢
>
>
>
>
>
> 在 2019年9月3日,11:44,王刚 <mail@wanggang1987.com> 写道:
>
>
>
> Hi All
>
>       我是苏宁财务平台的研发,我们在财务指标平台升级计划中正在考虑平台选型,kylin作为考察目标之一。
>
>       在目前的测试步骤中,遇到了事实表历史数据更新的问题,请教一下各位developer。
>
>       举例hive事实表orders,以创建时间作为时间分区
>
> 订单号
>
> 创建时间
>
> 支付时间
>
> 开票时间
>
> Order1
>
> 20190701
>
> 20190801
>
> 20190901
>
> Order2
>
> 20190901
>
> 20190901
>
> 20190901
>
>
>
>       表中order2的时间分区一致且不更新,比较容易计算cube。在order1这条数据中,20190801和20190901
> 分别对时间分区为20190701的数据支付时间和开票时间做了更新,因此支付和开票相关的统计指标也需要更新。
>
>       请问如何配置cube和增量更新方式,能够最高效实现事实表历史数据的指标更新呢?
>
>       kylin新手,诚心请教,烦请各位不吝赐家,多谢。
>
>
>
> ****************************************************************************
> Email Disclaimer
> ----------------
> The information in this email is confidential and may be legally
> privileged.
> It is intended solely for the addressee. Access to this email by anyone
> else
> is unauthorised.
>
> If you have received this email in error, please reply to the sender and
> notify him/her of this and delete the email (including all its enclosures)
> and destroy all copies of it. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>
> Any opinions, conclusions, advice or statements contained in this email
> should not be relied upon unless they are confirmed in writing on hard copy
> letterhead. Opinions, conclusions and other information in this email and
> any attachments that do not relate to the official business of the firm are
> neither given nor endorsed by it.
>
> There is no guarantee that email communications are secure or error-free,
> as information could be intercepted, corrupted, amended, lost, destroyed,
> arrive late or incomplete, or contain viruses.
>
> KPMG, a Hong Kong partnership and KPMG Huazhen LLP, a People's Republic
> of China partnership, are member firms of the KPMG network of independent
> member firms affiliated with KPMG International Cooperative.
>
> KPMG and KPMG Huazhen LLP, together with various
> affiliated entities operate in the People's Republic of China including
> Hong Kong SAR and Macau SAR, as KPMG China.
>
>
> KPMG International Cooperative is a Swiss entity. Member firms of the KPMG
> network of independent firms are affiliated with KPMG International
> Cooperative.
> KPMG International Cooperative provides no client services.
>
> Web sites: kpmg.com/cn
>
> *****************************************************************************
>
Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message