hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6436) JobHistory cache issue
Date Fri, 17 Jul 2015 07:15:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630890#comment-14630890
] 

Hadoop QA commented on MAPREDUCE-6436:
--------------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  16m 18s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear to include any
new or modified tests.  Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. |
| {color:green}+1{color} | javac |   8m  4s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |  10m 12s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   0m 30s | The applied patch generated  1 new checkstyle
issues (total was 16, now 17). |
| {color:red}-1{color} | whitespace |   0m  0s | The patch has 1  line(s) that end in whitespace.
Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with eclipse:eclipse.
|
| {color:green}+1{color} | findbugs |   0m 53s | The patch does not introduce any new Findbugs
(version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   5m 53s | Tests passed in hadoop-mapreduce-client-hs.
|
| | |  44m 15s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12745774/MAPREDUCE-6436.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ee36f4f |
| checkstyle |  https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5894/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-hs.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5894/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-hs test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5894/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5894/testReport/
|
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5894/console |


This message was automatically generated.

> JobHistory cache issue
> ----------------------
>
>                 Key: MAPREDUCE-6436
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6436
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Ryu Kobayashi
>            Assignee: Ryu Kobayashi
>         Attachments: MAPREDUCE-6436.1.patch, stacktrace1.txt, stacktrace2.txt, stacktrace3.txt
>
>
> Problem: 
> HistoryFileManager.addIfAbsent produces large amount of logs if number of
> cached entries whose age is less than mapreduce.jobhistory.max-age-ms becomes
> larger than mapreduce.jobhistory.joblist.cache.size by far.
> Example:
> For example, if the cache contains 50000 entries in total and 10,000 entries
> newer than mapreduce.jobhistory.max-age-ms where
> mapreduce.jobhistory.joblist.cache.size is 20000, HistoryFileManager.addIfAbsent
> method produces 50000 - 20000 = 30000 lines of "Waiting to remove <key> from
> JobListCache because it is not in done yet" message.
> It will attach a stacktrace.
> Impact:
> In addition to large disk consumption, this issue blocks JobHistory.getJob
> long time and slows job execution down significantly because getJob is called
> by RPC such as HistoryClientService.HSClientProtocolHandler.getJobReport.
> This impact happens because HistoryFileManager.UserLogDir.scanIfNeeded
> eventually calls HistoryFileManager.addIfAbsent in a synchronized block. When
> multiple threads call scanIfNeeded simultaneously, one of them acquires lock
> and the other threads are blocked until the first thread completes long-running
> HistoryFileManager.addIfAbsent call.
> Solution: 
> * Reduce amount of logs so that HistoryFileManager.addIfAbsent doesn't take too long
time.
> * Good to have if possible: HistoryFileManager.UserLogDir.scanIfNeeded skips
>   scanning if another thread is already scanning. This changes semantics of
>   some HistoryFileManager methods (such as getAllFileInfo and getFileInfo)
>   because scanIfNeeded keep outdated state.
> * Good to have if possible: Make scanIfNeeded asynchronous so that RPC calls are
>   not blocked by a loop at scale of tens of thousands.
>  
> This patch implemented the first item.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message