hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5470) LocalJobRunner does not work on Windows.
Date Thu, 22 Aug 2013 06:58:52 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747307#comment-13747307
] 

Chris Nauroth commented on MAPREDUCE-5470:
------------------------------------------

Sandy, thanks so much for taking a look!

bq. So either should move the task dirs under localJobDir or change back the part related
to task dirs. It's not obvious to me which is better.

I believe it should be the latter, because there are different assumptions about retention
of files in the local job dir vs. the task attempt directories.  For example, users expect
to be able to control retention of task files by setting {{mapreduce.task.files.preserve.failedtasks}}
or {{mapreduce.task.files.preserve.filepattern}}.  Otherwise, they expect task attempt files
to get cleaned up promptly.  (Please check my thinking on this though and let me know what
you think.)

                
> LocalJobRunner does not work on Windows.
> ----------------------------------------
>
>                 Key: MAPREDUCE-5470
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5470
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Chris Nauroth
>
> {{LocalJobRunner#getLocalTaskDir}} creates a directory that is unique to the task ID.
 The logic of this method concatenates the local job dir and a task-specific path, but one
of the arguments is a {{Path}} with a scheme, so the final result has "file:" embedded in
it.  This works on Linux, but the ':' is an invalid character in a file name on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message