hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (MAPREDUCE-317) Submitting job information via DFS in Map/Reduce causing consistency and performance issues
Date Mon, 21 Jul 2014 21:14:42 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Allen Wittenauer resolved MAPREDUCE-317.
----------------------------------------

    Resolution: Incomplete

With YARN, this isn't too relevant anymore. Closing as stale.

> Submitting job information via DFS in Map/Reduce causing consistency and performance
issues
> -------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-317
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-317
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Hemanth Yamijala
>
> Job submission involves two steps: submitting jobs to the System directory on DFS (done
by the client), then submit the job via the JobSubmissionProtocol to JobTracker. This two
step process is seen to have some issues:
> - Since the files need to be read from DFS, slowness in the DFS can cause job initialization
to become costly. We faced this as described in HADOOP-5286 and HADOOP-4664.
> - The two step process could lead to inconsistent information being left around - like
in HADOOP-5327 and HADOOP-5335.
> This JIRA is to explore options to remove the two step process in submitting a job.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message