kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhong Yanghong (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KYLIN-2866) Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
Date Sun, 03 Dec 2017 12:20:01 GMT

     [ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zhong Yanghong updated KYLIN-2866:
----------------------------------
    Fix Version/s: v2.3.0

> Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
> ------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-2866
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2866
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>            Reporter: Zhong Yanghong
>            Assignee: Zhong Yanghong
>             Fix For: v2.3.0
>
>
> Currently only one reducer is assigned for hll stats calculation, which may become the
bottleneck for slow down this step. Since the stats for different cuboids will not influence
each other, it's better to divide the cuboid set into several and assign a reduce for each
subset.
> The strategy of this patch is to assign 100 cuboids into a subset. And there's a upper
limit of reducers for hll stats calculation. Currently it's 50.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message