hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5047) keep.failed.task.files=true causes job failure on secure clusters
Date Wed, 06 Mar 2013 02:22:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594251#comment-13594251
] 

Sandy Ryza commented on MAPREDUCE-5047:
---------------------------------------

localizeConfiguration is needed in the tasktracker in order to set task-specific configuration
options, but split.info does not need to be created at that time.  The patch moves the action
of writing out split.info into a new writeFilesRequiredForRerun method.  This method is called
by the Child, but not by the tasktracker.

Tested on a pseudo distributed cluster and on a secure distributed cluster that the permissions
error no longer shows up and that split.info is still written out to the correct location.
                
> keep.failed.task.files=true causes job failure on secure clusters
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-5047
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5047
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task, tasktracker
>    Affects Versions: 1.1.1
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: MAPREDUCE-5047.patch
>
>
> To support IsolationRunner, split info is written to local directories.  This occurs
inside MapTask#localizeConfiguration, which is called both tasktracker and by the child JVM.
 On a secure cluster, the tasktacker's attempt to write it fails, because the tasktracker
does not have permission to write to the user's directory. It is likely that the call to localizeConfiguration
in the tasktracker can be removed. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message