flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1139) HadoopOutputFormat fails for DOP > 1
Date Fri, 21 Nov 2014 11:09:33 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220777#comment-14220777
] 

ASF GitHub Bot commented on FLINK-1139:
---------------------------------------

Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/incubator-flink/pull/173#discussion_r20709227
  
    --- Diff: flink-core/src/main/java/org/apache/flink/api/common/operators/GenericDataSinkBase.java
---
    @@ -292,13 +294,23 @@ public void accept(Visitor<Operator<?>> visitor) {
     	
     	protected void executeOnCollections(List<IN> inputData) throws Exception {
     		OutputFormat<IN> format = this.formatWrapper.getUserCodeObject();
    +		
    +		if(format instanceof InitializeOnMaster) {
    --- End diff --
    
    Good catch!


> HadoopOutputFormat fails for DOP > 1
> ------------------------------------
>
>                 Key: FLINK-1139
>                 URL: https://issues.apache.org/jira/browse/FLINK-1139
>             Project: Flink
>          Issue Type: Bug
>          Components: Hadoop Compatibility
>            Reporter: Fabian Hueske
>            Assignee: Fabian Hueske
>
> HadoopOutputFormat fails for DOP > 1. The reason is that job finalization (removal
of the temp directory) is done after the first task finishes.
> All other tasks will fail in that moment, since they cannot continue to write their data
to the temp directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message