hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
Date Fri, 04 Mar 2016 18:55:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180367#comment-15180367

Kihwal Lee commented on MAPREDUCE-6527:

LocalJobRunner is for testing map reduce locally without involving the actual cluster. Impact
of the race is minimal. If the input or output path is in a secure HDFS, it might cause the
local job instance to fail. If the job uses the local file system or a HDFS with security
disabled, there will be no issue.

> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-6527
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.7.1
>            Reporter: Ali Kheradmand
>            Assignee: Haibo Chen
>         Attachments: mapreduce6527.001.patch
> I am running the test suite against a dynamic race detector called RV-Predict. Here is
a race report that I got: 
> {noformat}
> Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
>     Concurrent read in thread T327 (locks held: {})
>  ---->  at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
>         at org.apache.hadoop.mapreduce.task.JobContextImpl.<init>(JobContextImpl.java:70)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
>     T327 is created by T22
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:218)
>     Concurrent write in thread T22 (locks held: {Monitor@496c673a, Monitor@496319b0})
>  ---->  at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
>         at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
>         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
>         at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>          locked Monitor@496319b0 at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)

>         at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
>          locked Monitor@496c673a at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)

>     T22 is created by T1
>         at org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
> }}}
> {noformat}
> In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob function, we have
the following lines:
> {code}
>     Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
>     job.job.setCredentials(credentials);
> {code}
> It looks a bit suspicious: Job extends thread and at the end of its constructor it starts
a new thread which creates a new instance of JobContextImpl which reads credentials. However,
the first thread concurrently sets credentials after a creating the Job instance. 

This message was sent by Atlassian JIRA

View raw message