zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-1669) Operations to server will be timed-out while thousands of sessions expired same time
Date Mon, 31 Jul 2017 02:01:09 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106730#comment-16106730
] 

Hadoop QA commented on ZOOKEEPER-1669:
--------------------------------------

+1 overall.  GitHub Pull Request  Build
      

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require
tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 3.0.1) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit
warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/910//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/910//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/910//console

This message is automatically generated.

> Operations to server will be timed-out while thousands of sessions expired same time
> ------------------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-1669
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1669
>             Project: ZooKeeper
>          Issue Type: Improvement
>          Components: server
>    Affects Versions: 3.3.5
>            Reporter: tokoot
>            Assignee: Cheney Sun
>              Labels: performance
>
> If there are thousands of clients, and most of them disconnect with server same time(client
restarted or servers partitioned with clients), the server will busy to close those "connections"
and become unavailable. The problem is in following:
>   private void closeSessionWithoutWakeup(long sessionId) {
>       HashSet<NIOServerCnxn> cnxns;
>           synchronized (this.cnxns) {
>               cnxns = (HashSet<NIOServerCnxn>)this.cnxns.clone();  // other thread
will block because of here
>           }
>       ...
>   }
> A real world example that demonstrated this problem (Kudos to [~sun.cheney]):
> {noformat}
> The issue is raised while tens thousands of clients try to reconnect ZooKeeper service.

> Actually, we came across the issue during maintaining our HBase cluster, which used a
5-server ZooKeeper cluster. 
> The HBase cluster was composed of many many regionservers (in thousand order of magnitude),

> and connected by tens thousands of clients to do massive reads/writes. 
> Because the r/w throughput is very high, ZooKeeper zxid increased quickly as well. 
> Basically, each two or three weeks, Zookeeper would make leader relection triggered by
the zxid roll over. 
> The leader relection will cause the clients(HBase regionservers and HBase clients) disconnected

> and reconnected with Zookeeper servers in the mean time, and try to renew the sessions.
> In current implementation of session renew, NIOServerCnxnFactory will clone all the connections
at first 
> in order to avoid race condition in multi-threads and go iterate the cloned connection
set one by one to 
> find the related session to renew. It's very time consuming. In our case (described above),

> it caused many region servers can't successfully renew session before session timeout,

> and eventually the HBase cluster lose these region servers and affect the HBase stability.
> The change is to make refactoring to the close session logic and introduce a ConcurrentHashMap

> to store session id and connection map relation, which is a thread-safe data structure

> and eliminate the necessary to clone the connection set at first.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message