hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Stack (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (HBASE-23919) [Flakey Test] Standalone Zookeeper won't start (minizookeepercluster won't come up)
Date Tue, 24 Mar 2020 05:22:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-23919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Stack resolved HBASE-23919.
-----------------------------------
    Resolution: Not A Problem

HBASE-23993 fixed this condition

> [Flakey Test] Standalone Zookeeper won't start (minizookeepercluster won't come up)
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-23919
>                 URL: https://issues.apache.org/jira/browse/HBASE-23919
>             Project: HBase
>          Issue Type: Bug
>          Components: flakies
>            Reporter: Michael Stack
>            Priority: Major
>
> I've seen this on occasion across different hardwares; the standalone zookeeper won't
come up and then random unit test fails in its junit startup phase as part of launching mini
cluster.
> I've been trying to track this w/ a while now adding in logging, using more of zk client
instead of the copy/paste we've had a long while now, and adding in logging (I've been running
locally w/ zk logging set to INFO).
> It looks like this currently where the last thing out of the server launch is this....
> {code:java}
>   2020-03-02 07:57:46,129 INFO  [Time-limited test] server.ZooKeeperServer(854): maxSessionTimeout
set to -1                                                                                
                                           2020-03-02 07:57:46,139 INFO  [Time-limited test]
server.NIOServerCnxnFactory(89): binding to port 0.0.0.0/0.0.0.0:49316                   
                                                                                         
2020-03-02 07:57:46,181 INFO  [Time-limited test] zookeeper.MiniZooKeeperCluster(256): Started
connectionTimeout=30000, dir=/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/89d57393-fe97-9200-630f-7843ee406bd2/
     cluster_6b4d6f67-7978-dc67-a1a3-7b1b0c0e4268/zookeeper_0, clientPort=49316, dataDir=/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/89d57393-fe97-9200-630f-7843ee406bd2/cluster_6b4d6f67-7978-dc67-a1a3-7b1b0c0e4268/
zookeeper_0/version-2, dataLogDir=/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/89d57393-fe97-9200-630f-7843ee406bd2/cluster_6b4d6f67-7978-dc67-a1a3-7b1b0c0e4268/zookeeper_0/version-2,
tickTime=2000,              maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000,
serverId=0{code}
> ... then the client just does this over and over:
> {code:java}
>  2020-03-02 07:57:46,182 INFO  [Time-limited test] client.FourLetterWordMain(65): connecting
to localhost 49316                                                                       
                                                2020-03-02 07:57:46,213 INFO  [Time-limited
test] zookeeper.MiniZooKeeperCluster(453): localhost:49316 not up                        
                                                                                         
      java.net.SocketException: Connection reset                                         
                                                                                         
                                                          at java.net.SocketInputStream.read(SocketInputStream.java:209)
                                                                                         
                                                                             at java.net.SocketInputStream.read(SocketInputStream.java:141)
                                                                                         
                                                                             at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
                                                                                         
                                                                              at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
                                                                                         
                                                                               at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
                                                                                         
                                                                                   at java.io.InputStreamReader.read(InputStreamReader.java:184)
                                                                                         
                                                                              at java.io.BufferedReader.fill(BufferedReader.java:161)
                                                                                         
                                                                                    at java.io.BufferedReader.readLine(BufferedReader.java:324)
                                                                                         
                                                                                at java.io.BufferedReader.readLine(BufferedReader.java:389)
                                                                                         
                                                                                at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:84)
                                                                                         
                                              at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.waitForServerUp(MiniZooKeeperCluster.java:442)
                                                                                         
                                   at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259)
                                                                                         
                                           at org.apache.hadoop.hbase.HBaseZKTestingUtility.startMiniZKCluster(HBaseZKTestingUtility.java:130)
                                                                                         
                                        at org.apache.hadoop.hbase.HBaseZKTestingUtility.startMiniZKCluster(HBaseZKTestingUtility.java:103)
                                                                                         
                                        at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1107)
                                                                                         
                                             at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1065)
                                                                                         
                                             at org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable.before(TestRegionReplicasWithModifyTable.java:62)
                                                                                         
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)           
                                                                                         
                                                                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                                                                         
                                                           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                                                                         
                                                   at java.lang.reflect.Method.invoke(Method.java:498)
                                                                                         
                                                                                        at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)     
                                                                                         
                                               at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
                                                                                         
                                                     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
                                                                                         
                                                      at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
                                                                                         
                                                       at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
                                                                                         
                                                           at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
                                                                                         
                                                             at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
                                                                                         
                                      at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
                                                                                         
                                      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                                                                                         
                                                                                at java.lang.Thread.run(Thread.java:745)
>  {code}
>  
> There is ZOOKEEPER-2714 but it is closed as not-reproducible and it looks a little different
going by the log emissions in that it registers the client connections.
> Noting this here in case others have seen similar or there are ideas out there for how
to fix.
> Here for example is how the failure might look high-level:
> {code:java}
>  -------------------------------------------------------------------------------
>  Test set: org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable
>  -------------------------------------------------------------------------------
>  Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.017 s <<<
FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable
>  org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable  Time elapsed:
0.01 s  <<< ERROR!
>  java.io.IOException: Waiting for startup of standalone server; server isRunning=true
>    at 
> org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable.before(TestRegionReplicasWithModifyTable.java:62)
{code}
> ... and then when you dig in, the test never even made it past cluster launch in setup. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message