kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dong Li (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KYLIN-2866) Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
Date Thu, 21 Dec 2017 03:28:00 GMT

     [ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dong Li resolved KYLIN-2866.
----------------------------
    Resolution: Fixed

Thanks Yanghong, patch has been merged to master branch.

> Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
> ------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-2866
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2866
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>            Reporter: Zhong Yanghong
>            Assignee: Zhong Yanghong
>             Fix For: v2.3.0
>
>         Attachments: APACHE-KYLIN-2866-refined.patch, APACHE-KYLIN-2866.patch
>
>
> Currently only one reducer is assigned for hll stats calculation, which may become the
bottleneck for slow down this step. Since the stats for different cuboids will not influence
each other, it's better to divide the cuboid set into several and assign a reduce for each
subset.
> The strategy of this patch is to assign 100 cuboids into a subset. And there's a upper
limit of reducers for hll stats calculation. Currently it's 50.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message