hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitesh Shah (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
Date Mon, 12 Dec 2011 22:37:30 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167894#comment-13167894

Hitesh Shah commented on MAPREDUCE-3426:

Summary of changes: 

LocalContainerLauncher changes:
  - in runSubTask(), set the correct config values for task specific information as well as
the local dirs setting so that the output is written to the correct dir ( which the process
has write permissions to )
  - cleaned up commented out code. 

Added flags to be able to profile map and reduce tasks with separate profiling params. 

MRAppMaster changes: 
  - fixed disabling of speculative execution if job is uberized.
  - Moved uber decision into new function
    - hardcoded max reduce count supported to 1 as multiple reduces are not supported. 
    - using MR AM vm size as memory bound check for uber decision 
RMCommunicator change:
  - for MR-3421 

JobStatus/JobReport changes:
  - Print out uber choice on client side to inform user whether running job in uber or non-uber

Comments changes:
   - Removed comments on race condition in local launcher for "MapTask can't send TA_UPDATE
     - the state machine has a write lock, also the launched event is sent first followed
by any updates if any from the task  itself. 
   - renameMapOutputForReduce is still required. Output files are generated into top level
dir and not a task specific dir. 
> uber-jobs tried to write outputs into wrong dir
> -----------------------------------------------
>                 Key: MAPREDUCE-3426
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2
>    Affects Versions: 0.23.0, 0.24.0
>            Reporter: Hitesh Shah
>             Fix For: 0.23.1
>         Attachments: MR-3426.1.patch, MR-3426.2.patch
> Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into
dirs that the user does not have permissions to write to on a secure cluster. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message