hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3591) Resource Localisation on a bad disk causes subsequent containers failure
Date Wed, 13 May 2015 07:33:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541525#comment-14541525
] 

Hadoop QA commented on YARN-3591:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  14m 40s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear to include any
new or modified tests.  Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. |
| {color:green}+1{color} | javac |   7m 35s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |   9m 34s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   0m 35s | The applied patch generated  2 new checkstyle
issues (total was 19, now 20). |
| {color:red}-1{color} | whitespace |   0m  0s | The patch has 2  line(s) that end in whitespace.
Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with eclipse:eclipse.
|
| {color:green}+1{color} | findbugs |   1m  2s | The patch does not introduce any new Findbugs
(version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests |   6m  0s | Tests passed in hadoop-yarn-server-nodemanager.
|
| | |  41m 59s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12732494/YARN-3591.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fcd0702 |
| checkstyle |  https://builds.apache.org/job/PreCommit-YARN-Build/7913/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/7913/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7913/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7913/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7913/console |


This message was automatically generated.

> Resource Localisation on a bad disk causes subsequent containers failure 
> -------------------------------------------------------------------------
>
>                 Key: YARN-3591
>                 URL: https://issues.apache.org/jira/browse/YARN-3591
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Lavkesh Lahngir
>         Attachments: 0001-YARN-3591.1.patch, 0001-YARN-3591.patch, YARN-3591.2.patch
>
>
> It happens when a resource is localised on the disk, after localising that disk has gone
bad. NM keeps paths for localised resources in memory.  At the time of resource request isResourcePresent(rsrc)
will be called which calls file.exists() on the localised path.
> In some cases when disk has gone bad, inodes are stilled cached and file.exists() returns
true. But at the time of reading, file will not open.
> Note: file.exists() actually calls stat64 natively which returns true because it was
able to find inode information from the OS.
> A proposal is to call file.list() on the parent path of the resource, which will call
open() natively. If the disk is good it should return an array of paths with length at-least
1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message