flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefano Bortoli (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2394) HadoopOutFormat OutputCommitter is default to FileOutputCommiter
Date Wed, 22 Jul 2015 15:27:04 GMT
Stefano Bortoli created FLINK-2394:

             Summary: HadoopOutFormat OutputCommitter is default to FileOutputCommiter
                 Key: FLINK-2394
                 URL: https://issues.apache.org/jira/browse/FLINK-2394
             Project: Flink
          Issue Type: Bug
          Components: Hadoop Compatibility
    Affects Versions: 0.9.0
            Reporter: Stefano Bortoli

MongoOutputFormat does not write back in collection because the HadoopOutputFormat wrapper
does not allow to set the MongoOutputCommiter and is set as default to FileOutputCommitter.
Therefore, on close and globalFinalize execution the commit does not happen and mongo collection
stays untouched. 

A simple solution would be to:

1 - create a constructor of HadoopOutputFormatBase and HadoopOutputFormat that gets the OutputCommitter
as a parameter
2 - change the outputCommitter field of HadoopOutputFormatBase to be a generic OutputCommitter
3 - remove the default assignment in the open() and finalizeGlobal to the outputCommitter
to FileOutputCommitter(), or keep it as a default in case of no specific assignment.

This message was sent by Atlassian JIRA

View raw message