hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13317) Region server reportForDuty stuck looping if there is a master change
Date Mon, 23 Mar 2015 19:42:53 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376480#comment-14376480
] 

Hadoop QA commented on HBASE-13317:
-----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12706577/HBASE-13317-0.98.patch
  against 0.98 branch at commit 845f5de121e92a99b41b30dc86cb3f2898e0254f.
  ATTACHMENT ID: 12706577

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions
(2.4.1 2.5.2 2.6.0)

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 25 warning messages.

    {color:green}+1 checkstyle{color}.  The applied patch does not increase the total number
of checkstyle errors

    {color:red}-1 findbugs{color}.  The patch appears to introduce 8 new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

     {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s): 	at org.apache.maven.shared.test.plugin.BuildTool.executeMaven(BuildTool.java:93)
	at org.apache.maven.plugin.eclipse.it.AbstractEclipsePluginIT.testProject(AbstractEclipsePluginIT.java:333)
	at org.apache.maven.plugin.eclipse.it.AbstractEclipsePluginIT.testProject(AbstractEclipsePluginIT.java:290)
	at org.apache.maven.plugin.eclipse.it.AbstractEclipsePluginIT.testProject(AbstractEclipsePluginIT.java:272)
	at org.apache.maven.plugin.eclipse.it.AbstractEclipsePluginIT.testProject(AbstractEclipsePluginIT.java:245)
	at org.apache.maven.plugin.eclipse.it.EclipsePluginIT.testProject65(EclipsePluginIT.java:767)

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13369//console

This message is automatically generated.

> Region server reportForDuty stuck looping if there is a master change
> ---------------------------------------------------------------------
>
>                 Key: HBASE-13317
>                 URL: https://issues.apache.org/jira/browse/HBASE-13317
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 1.0.0, 2.0.0, 0.98.12
>            Reporter: Jerry He
>            Assignee: Jerry He
>             Fix For: 2.0.0, 1.0.1, 0.98.13
>
>         Attachments: HBASE-13317-0.98-v2.patch, HBASE-13317-0.98.patch
>
>
> During cluster startup, region server reportForDuty gets stuck looping if there is a
master change.
> {noformat}
> 2015-03-22 11:15:16,186 INFO  [regionserver60020] regionserver.HRegionServer: reportForDuty
to master=bigaperf274,60000,1427045883965 with port=60020, startcode=1427048115174
> 2015-03-22 11:15:16,272 WARN  [regionserver60020] regionserver.HRegionServer: error telling
master we are up
> com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
> 	at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
> 	at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
> 	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-03-22 11:15:16,274 WARN  [regionserver60020] regionserver.HRegionServer: reportForDuty
failed; sleeping and then retrying.
> 2015-03-22 11:15:19,274 INFO  [regionserver60020] regionserver.HRegionServer: reportForDuty
to master=bigaperf273,60000,1427048108439 with port=60020, startcode=1427048115174
> 2015-03-22 11:15:19,275 WARN  [regionserver60020] regionserver.HRegionServer: error telling
master we are up
> com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
> 	at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
> 	at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
> 	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-03-22 11:15:19,276 WARN  [regionserver60020] regionserver.HRegionServer: reportForDuty
failed; sleeping and then retrying.
> 2015-03-22 11:15:22,276 INFO  [regionserver60020] regionserver.HRegionServer: reportForDuty
to master=bigaperf273,60000,1427048108439 with port=60020, startcode=1427048115174
> 2015-03-22 11:15:22,296 DEBUG [regionserver60020] regionserver.HRegionServer: Master
is not running yet
> 2015-03-22 11:15:22,296 WARN  [regionserver60020] regionserver.HRegionServer: reportForDuty
failed; sleeping and then retrying.
> 2015-03-22 11:15:25,296 INFO  [regionserver60020] regionserver.HRegionServer: reportForDuty
to master=bigaperf273,60000,1427048108439 with port=60020, startcode=1427048115174
> 2015-03-22 11:15:25,299 DEBUG [regionserver60020] regionserver.HRegionServer: Master
is not running yet
> 2015-03-22 11:15:25,299 WARN  [regionserver60020] regionserver.HRegionServer: reportForDuty
failed; sleeping and then retrying.
> 2015-03-22 11:15:28,299 INFO  [regionserver60020] regionserver.HRegionServer: reportForDuty
to master=bigaperf273,60000,1427048108439 with port=60020, startcode=1427048115174
> 2015-03-22 11:15:28,302 DEBUG [regionserver60020] regionserver.HRegionServer: Master
is not running yet
> 2015-03-22 11:15:28,302 WARN  [regionserver60020] regionserver.HRegionServer: reportForDuty
failed; sleeping and then retrying.
> {noformat}
> What happended is the region server first got master=bigaperf274,60000,1427045883965.
 Before it was able to report successfully, the maser changed to bigaperf273,60000,1427048108439.
> We were supposed to open a new connection to the new master. But we never did, looping
and trying to old address forever.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message