hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Marz (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-5975) Provide more hooks into OutputFormat/RecordWriter events
Date Thu, 04 Jun 2009 23:15:07 GMT
Provide more hooks into OutputFormat/RecordWriter events

                 Key: HADOOP-5975
                 URL: https://issues.apache.org/jira/browse/HADOOP-5975
             Project: Hadoop Core
          Issue Type: Improvement
    Affects Versions: 0.18.3
            Reporter: Nathan Marz

OutputFormat could be made a lot more flexible if more hooks were added for various events
that occur during a job. In particular, I would like to see:

1. RecordWriter#cleanup : called whether the task succeeds, fails, or gets killed. I have
an OutputFormat which writes to local disk but has no way of cleaning up on failure or getting

2. OutputFormat#close - Once the job has completely run, I sometimes need the OutputFormat
to do some cleanup or additional logic to finalize the output. Right now I have to stuff this
logic into Cascading Taps but things would work better and be more flexible if I could put
it in the OutputFormat itself.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message