hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1802) allow outputcommitters to skip setup/cleanup
Date Thu, 20 May 2010 01:01:55 GMT
allow outputcommitters to skip setup/cleanup
--------------------------------------------

                 Key: MAPREDUCE-1802
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1802
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Joydeep Sen Sarma
            Assignee: Joydeep Sen Sarma


Job setup and cleanup overheads in our (larger) clusters are very significant and add to latency
for small jobs. It turns out that Hive does not require job setup and cleanup at all - since
all management of output/temporary files and such is done by the hive client side. So it would
be a big win for our environment (and Hive users in general) if we could skip job cleanup/setup
altogether.

The proposal is to add new calls to OutputCommitter interface (along the lines of needsTaskCommit())
to optionally allow skipping of setup/cleanup and for the JT to take these into account while
scheduling setup/cleanup. NullOutputFormat should not need setup/cleanup for example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message