hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
Date Thu, 03 Mar 2016 22:11:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178703#comment-15178703

Haibo Chen commented on MAPREDUCE-6527:

I believe the expected behavior for accessing the file system without proper credential is
denial of access, so I don't think this bug will allow the thread to access files that it
doesn't have access to. The credentials are used by the output committer to make commit directories
for local jobs, therefore, I think it could potentially cause directory creations to fail.
I am not quite sure about the security setup of FileSystem in LocalJobRunner, it could totally
be possible that the security is not enabled/enforced by the File System. Anyone knows some
details of this?

> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-6527
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.7.1
>            Reporter: Ali Kheradmand
>            Assignee: Haibo Chen
>         Attachments: mapreduce6527.001.patch
> I am running the test suite against a dynamic race detector called RV-Predict. Here is
a race report that I got: 
> {noformat}
> Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
>     Concurrent read in thread T327 (locks held: {})
>  ---->  at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
>         at org.apache.hadoop.mapreduce.task.JobContextImpl.<init>(JobContextImpl.java:70)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
>     T327 is created by T22
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:218)
>     Concurrent write in thread T22 (locks held: {Monitor@496c673a, Monitor@496319b0})
>  ---->  at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
>         at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
>         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
>         at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>          locked Monitor@496319b0 at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)

>         at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
>          locked Monitor@496c673a at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)

>     T22 is created by T1
>         at org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
> }}}
> {noformat}
> In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob function, we have
the following lines:
> {code}
>     Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
>     job.job.setCredentials(credentials);
> {code}
> It looks a bit suspicious: Job extends thread and at the end of its constructor it starts
a new thread which creates a new instance of JobContextImpl which reads credentials. However,
the first thread concurrently sets credentials after a creating the Job instance. 

This message was sent by Atlassian JIRA

View raw message