hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-7015) Possible race condition in JHS if the job is not loaded
Date Wed, 03 Jan 2018 15:02:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309767#comment-16309767
] 

Jason Lowe commented on MAPREDUCE-7015:
---------------------------------------

IIRC one reason the moveToDoneExecutor was added was to avoid spending a long time processing
every move necessary during the scan.  By adding the sync parameter or removing the executor
completely, that makes the scan move everything it finds inline. This can take quite a long
time since it not only has to wait for RPC calls but also can spend a long time waiting to
acquire the lock because another thread could be calling loadJob() at the time and waiting
on a very slow datanode.  That makes the scan single-threaded since it can't make progress
on other intermediate files until it finishes moving each one in order.  So I don't think
making this sync is the ideal solution.

It may make more sense to have the RPC call wait for out-of-band intermediate scan results
it will return to be moved rather than forcing the entire scan to be single-threaded or always
waiting for all intermediates to be moved.  For example, HistoryFileManager#getFileInfo could
explicitly wait on the move to complete for the one job it is interested in if that job was
found in the intermediate scan.  Then we don't have to wait for _every_ intermediate job to
be moved, just the one we care about.  getAllFileInfo would need to wait for all of them,
but they would be processed in parallel while we're waiting for each in turn.

> Possible race condition in JHS if the job is not loaded
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-7015
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7015
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>         Attachments: MAPREDUCE-7015-POC01.patch
>
>
> There could be a race condition inside JHS. In our build environment, {{TestMRJobClient.testJobClient()}}
failed with this exception:
> {noformat}
> ava.io.FileNotFoundException: File does not exist: hdfs://localhost:32836/tmp/hadoop-yarn/staging/history/done_intermediate/jenkins/job_1509975084722_0001_conf.xml
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2123)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2092)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2068)
> 	at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:460)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.mapreduce.TestMRJobClient.runTool(TestMRJobClient.java:94)
> 	at org.apache.hadoop.mapreduce.TestMRJobClient.testConfig(TestMRJobClient.java:551)
> 	at org.apache.hadoop.mapreduce.TestMRJobClient.testJobClient(TestMRJobClient.java:167)
> {noformat}
> Root cause:
> 1. MapReduce job completes
> 2. CLI calls {{cluster.getJob(jobid)}}
> 3. The job is finished and the client side gets redirected to JHS
> 4. The job data is missing from {{CachedHistoryStorage}} so JHS tries to find the job
> 5. First it scans the intermediate directory and finds the job
> 6. The call {{moveToDone()}} is scheduled for execution on a separate thread inside {{moveToDoneExecutor}}
and it starts to run immediately
> 7. RPC invocation returns with the path pointing to {{/tmp/hadoop-yarn/staging/history/done_intermediate}}
> 8. The call to {{moveToDone()}} completes which moves the contents of {{done_intermediate}}
to {{done}}
> 9. Hadoop CLI tries to download the config file from done_intermediate but it's no longer
there
> Usually step #6 is slow enough to complete after #7, but sometimes it's faster, causing
this race condition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message