hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4467) Shell.checkIsBashSupported swallowed an interrupted exception
Date Fri, 28 Oct 2016 00:06:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613732#comment-15613732
] 

Hadoop QA commented on YARN-4467:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 16s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 15s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 40s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 51s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 27s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 23s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 26s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 59s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  4s{color} |
{color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 17s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m  3s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 51s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 51s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 31s{color}
| {color:green} root: The patch generated 0 new + 66 unchanged - 1 fixed = 66 total (was 67)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 30s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 33s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 23s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  8s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 15s{color} | {color:green}
hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 11s{color} | {color:green}
hadoop-yarn-server-nodemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 27s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 56s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-4467 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12778308/YARN-4467.001.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 5bc334ab1efc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 022bf78 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13602/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U: . |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13602/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Shell.checkIsBashSupported swallowed an interrupted exception
> -------------------------------------------------------------
>
>                 Key: YARN-4467
>                 URL: https://issues.apache.org/jira/browse/YARN-4467
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>            Priority: Blocker
>              Labels: oct16-easy, shell, supportability
>         Attachments: HADOOP-12652.001.patch, YARN-4467.001.patch
>
>
> Edit: move this JIRA from HADOOP to YARN, as Shell.checkIsBashSupported() is used, and
only used in YARN.
> Shell.checkIsBashSupported() creates a bash shell command to verify if the system supports
bash. However, its error message is misleading, and the logic should be updated.
> If the shell command throws an IOException, it does not imply the bash did not run successfully.
If the shell command process was interrupted, its internal logic throws an InterruptedIOException,
which is a subclass of IOException.
> {code:title=Shell.checkIsBashSupported|borderStyle=solid}
>     ShellCommandExecutor shexec;
>     boolean supported = true;
>     try {
>       String[] args = {"bash", "-c", "echo 1000"};
>       shexec = new ShellCommandExecutor(args);
>       shexec.execute();
>     } catch (IOException ioe) {
>       LOG.warn("Bash is not supported by the OS", ioe);
>       supported = false;
>     }
> {code}
> An example of it appeared in a recent jenkins job
> https://builds.apache.org/job/PreCommit-HADOOP-Build/8257/testReport/org.apache.hadoop.ipc/TestRPCWaitForProxy/testInterruptedWaitForProxy/
> The test logic in TestRPCWaitForProxy.testInterruptedWaitForProxy starts a thread, wait
it for 1 second, and interrupt the thread, expecting the thread to terminate. However, the
method Shell.checkIsBashSupported swallowed the interrupt, and therefore failed.
> {noformat}
> 2015-12-16 21:31:53,797 WARN  util.Shell (Shell.java:checkIsBashSupported(718)) - Bash
is not supported by the OS
> java.io.InterruptedIOException: java.lang.InterruptedException
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:930)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:838)
> 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1117)
> 	at org.apache.hadoop.util.Shell.checkIsBashSupported(Shell.java:716)
> 	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:705)
> 	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
> 	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:639)
> 	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
> 	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
> 	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:803)
> 	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:773)
> 	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:646)
> 	at org.apache.hadoop.ipc.RPC.waitForProtocolProxy(RPC.java:397)
> 	at org.apache.hadoop.ipc.RPC.waitForProtocolProxy(RPC.java:350)
> 	at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:330)
> 	at org.apache.hadoop.ipc.TestRPCWaitForProxy$RpcThread.run(TestRPCWaitForProxy.java:115)
> Caused by: java.lang.InterruptedException
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:503)
> 	at java.lang.UNIXProcess.waitFor(UNIXProcess.java:264)
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:920)
> 	... 15 more
> {noformat}
> The original design is not desirable, as it swallowed a potential interrupt, causing
TestRPCWaitForProxy.testInterruptedWaitForProxy to fail. Unfortunately, Java does not allow
this static method to throw exception. We should removed the static member variable, so that
the method can throw the interrupt exception. The node manager should call the static method,
instead of using the static member variable.
> This fix has an associated benefit: the tests could run faster, because it will no longer
need to spawn a bash process when it uses a Shell static method variable (which happens quite
often for checking what operating system Hadoop is running on)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message