hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-1106) LocalJobRunner should run with any file-system
Date Fri, 13 May 2011 03:22:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032811#comment-13032811
] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-1106:
----------------------------------------------------

bq. So, what is the use case that this JIRA attempts to address?
Back when I created this, I heard from some users about testing their jobs using LocalJobRunner(LJR)
on a sample data in an env almost similar to to the real cluster. Which was not possible as
LJR assumed local paths. One specific use case is testing a user's job on a small input but
with real distributed cache files on DFS; copying all of the remote files to local FS just
for testing is a little ugly.

Is that still a valid use case?

Heads up, in the MR next-gen branch, use cases like these are solved by running all the tasks
of the sample job in a single JVM.

> LocalJobRunner should run with any file-system
> ----------------------------------------------
>
>                 Key: MAPREDUCE-1106
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1106
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Rohan A Mehta
>              Labels: newbie
>         Attachments: MAPREDUCE-1106.patch, MAPREDUCE-1106.patch
>
>
> LocalJobRunner is hard-coded to run with only the local file-system. This will help users
write map/reduce programs in local mode yet accessing input/schema/data files etc. from any
file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message