flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Hogan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3480) Add hash-based strategy for ReduceFunction
Date Fri, 22 Jul 2016 20:44:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390164#comment-15390164

Greg Hogan commented on FLINK-3480:

Today I bumped into the performance discrepancy where a forwarding ship strategy can hurt
performance since we can only do a sorted reduce whereas with a partition hash we can use
the new hash-combiner.

What would be the spilling strategy for a hash-reducer and would this look much different
from using the hash-combiner followed by the sort-reducer?

> Add hash-based strategy for ReduceFunction
> ------------------------------------------
>                 Key: FLINK-3480
>                 URL: https://issues.apache.org/jira/browse/FLINK-3480
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Local Runtime
>            Reporter: Fabian Hueske
> This issue is related to FLINK-3477. 
> While FLINK-3477 proposes to add hash-based combine strategy for ReduceFunction, this
issue aims to add a hash-based strategy for the final aggregation.
> This will need again a special hash-table aggregation which allows for in-place updates
and append updates. However, it also needs to support spilling to disk in case of too tight
memory budgets.

This message was sent by Atlassian JIRA

View raw message