flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2655) Minimize intermediate merging of spilled buffers
Date Tue, 15 Sep 2015 12:51:46 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745387#comment-14745387

ASF GitHub Bot commented on FLINK-2655:

Github user asfgit closed the pull request at:


> Minimize intermediate merging of spilled buffers
> ------------------------------------------------
>                 Key: FLINK-2655
>                 URL: https://issues.apache.org/jira/browse/FLINK-2655
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime
>    Affects Versions: master
>            Reporter: Greg Hogan
> If the number of spilled buffers exceeds taskmanager.runtime.max-fan then the number
of files must reduced with an intermediate merge by reading, merging, and spilling into a
single, larger file.
> The current implementation performs an intermediate merge on all files. An optimal implementation
minimizes the amount of merged data by performing partial merges first.

This message was sent by Atlassian JIRA

View raw message