flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Hogan (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (FLINK-3480) Add hash-based strategy for ReduceFunction
Date Sat, 23 Jul 2016 02:09:20 GMT

     [ https://issues.apache.org/jira/browse/FLINK-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Greg Hogan updated FLINK-3480:
    Comment: was deleted

(was: Today I bumped into the performance discrepancy where a forwarding ship strategy can
hurt performance since we can only do a sorted reduce whereas with a partition hash we can
use the new hash-combiner.

What would be the spilling strategy for a hash-reducer and would this look much different
from using the hash-combiner followed by the sort-reducer?)

> Add hash-based strategy for ReduceFunction
> ------------------------------------------
>                 Key: FLINK-3480
>                 URL: https://issues.apache.org/jira/browse/FLINK-3480
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Local Runtime
>            Reporter: Fabian Hueske
> This issue is related to FLINK-3477. 
> While FLINK-3477 proposes to add hash-based combine strategy for ReduceFunction, this
issue aims to add a hash-based strategy for the final aggregation.
> This will need again a special hash-table aggregation which allows for in-place updates
and append updates. However, it also needs to support spilling to disk in case of too tight
memory budgets.

This message was sent by Atlassian JIRA

View raw message