hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19554) AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit
Date Fri, 22 Dec 2017 01:32:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300816#comment-16300816
] 

Duo Zhang commented on HBASE-19554:
-----------------------------------

Checked recent pre commit building, seems much better. And this a failure case

https://builds.apache.org/job/PreCommit-HBASE-Build/10609/artifact/patchprocess/patch-unit-hbase-server.txt

The error message
{quote}
[ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 157.505 s <<<
FAILURE! - in org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL
[ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL)  Time elapsed:
49.462 s  <<< ERROR!
java.lang.RuntimeException: 
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=11, exceptions:
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:59 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:00 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:02 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:06 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:16 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:26 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:36 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: 
Failed after attempts=11, exceptions:
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:58 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:19:59 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.io.IOException: Call to 604d085d7ec5/172.17.0.2:57745 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed
servers list: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:00 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:02 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:06 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:16 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:26 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Thu Dec 21 10:20:36 UTC 2017, RpcRetryingCaller{globalStartTime=1513851598131, pause=100,
maxAttempts=11}, java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on
connection exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745

Caused by: java.net.ConnectException: Call to 604d085d7ec5/172.17.0.2:57745 failed on connection
exception: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: org.apache.hadoop.hbase.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 604d085d7ec5/172.17.0.2:57745
Caused by: java.net.ConnectException: Connection refused
{quote}

The test output xml can not be generated which makes it really hard to find out the real problem...

> AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit
> --------------------------------------------------------------
>
>                 Key: HBASE-19554
>                 URL: https://issues.apache.org/jira/browse/HBASE-19554
>             Project: HBase
>          Issue Type: Bug
>          Components: Recovery, wal
>            Reporter: Duo Zhang
>            Assignee: Duo Zhang
>         Attachments: HBASE-19554.patch
>
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/10554/artifact/patchprocess/patch-unit-hbase-server.txt
> The error message is a bit strange:
> {quote}
> [ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL) Time elapsed:
20.627 s <<< ERROR!
> org.apache.hadoop.hbase.TableNotFoundException: Region of 'hbase:namespace,,1513320505933.451650152885a3b41d0b1110deca513c.'
is expected in the table of 'testThreeRSAbort', but hbase:meta says it is in the table of
'hbase:namespace'. hbase:meta might be damaged.
> {quote}
> It fails for both FSHLog and AsyncFSWAL. Need to dig more.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message