hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-5350) Submitting job information via DFS in Map/Reduce causing consistency and performance issues
Date Fri, 27 Feb 2009 14:01:15 GMT
Submitting job information via DFS in Map/Reduce causing consistency and performance issues
-------------------------------------------------------------------------------------------

                 Key: HADOOP-5350
                 URL: https://issues.apache.org/jira/browse/HADOOP-5350
             Project: Hadoop Core
          Issue Type: Improvement
          Components: mapred
            Reporter: Hemanth Yamijala


Job submission involves two steps: submitting jobs to the System directory on DFS (done by
the client), then submit the job via the JobSubmissionProtocol to JobTracker. This two step
process is seen to have some issues:

- Since the files need to be read from DFS, slowness in the DFS can cause job initialization
to become costly. We faced this as described in HADOOP-5286 and HADOOP-4664.
- The two step process could lead to inconsistent information being left around - like in
HADOOP-5327 and HADOOP-5335.

This JIRA is to explore options to remove the two step process in submitting a job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message