hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6858) HistoryFileManager thrashing due to high volume jobs
Date Wed, 08 Mar 2017 19:41:38 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901862#comment-15901862

Haibo Chen commented on MAPREDUCE-6858:

Thanks [~yufeigu] for reporting this issue. Which version did you see this issue?

> HistoryFileManager thrashing due to high volume jobs 
> -----------------------------------------------------
>                 Key: MAPREDUCE-6858
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6858
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>            Reporter: Yufei Gu
> JHS log shows that it tried to move the same *.jhist twice, and the second moving causes
> - JHS scans "done_intermediate" dir for files to process and adds them to a thread pool
> - Thread pool starts processing these files to move them to "done" dir
> - JHS scans "done_intermediate" again for files to process and adds them to a thread
> -- If we have enough jobs where the thread pool can't keep up with the scanning interval,
they'll get added twice (or more). If this keeps compounding,  jobs end up would pile up and
not getting processed for quite some time and getting lots of FileNotFoundException's.
> By default, it looks like the thread pool only has 3 threads in it (mapreduce.jobhistory.move.thread-count).
And the scan interval is 3 minutes (mapreduce.jobhistory.move.interval-ms). Perhaps we should
increase these?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message