hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Gao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
Date Wed, 07 Nov 2012 22:26:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492762#comment-13492762
] 

Yu Gao commented on MAPREDUCE-4396:
-----------------------------------

Attached ant test and test-patch result.

This patch can be verified by running unit test TestMRWithDistributedCache in a user directory
with restrictive permission (e.g. 700), so no new test is added.

Before this patch, TestMRWithDistributedCache#testLocalJobRunner could fail when the test
directory - test.build.data - is in a path where any ancestor doesn't have +x permissions
for others, with following exception:
java.io.FileNotFoundException: File build/test/mapred/local/archive/5659425182264546915_1878220893_1556512037/file/home/biadmin/Hadoop_upstream/branch-1/build/test/data/distributed.first
does not exist.
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
	at org.apache.hadoop.filecache.TestMRWithDistributedCache$DistributedCacheChecker.setup(TestMRWithDistributedCache.java:93)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

This results from private distributed cache not localized by LocalJobRunner. The patch adds
the localization for private distributed cache, so that no matter whether the test directory
permission is too restrictive or not, distributed cache (public/private) will be localized.
                
> Make LocalJobRunner work with private distributed cache
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-4396
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 1.0.3
>            Reporter: Luke Lu
>            Assignee: Yu Gao
>            Priority: Minor
>         Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result,
test-patch.result
>
>
> Some LocalJobRunner related unit tests fails if user directory permission and/or umask
is too restrictive.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message