zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chang Song (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-1049) Session expire/close flooding renders heartbeats to delay significantly
Date Sat, 16 Apr 2011 04:04:05 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020555#comment-13020555
] 

Chang Song commented on ZOOKEEPER-1049:
---------------------------------------

log4j.rootLogger=ERROR, ROLLINGFILE

log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=INFO
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n

log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=DEBUG
log4j.appender.ROLLINGFILE.File=/???/zookeeper/log/zookeeper.log

log4j.appender.ROLLINGFILE.MaxFileSize=50MB
log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n

log4j.appender.TRACEFILE=org.apache.log4j.FileAppender
log4j.appender.TRACEFILE.Threshold=TRACE
log4j.appender.TRACEFILE.File=/???/zookeeper/log/zookeeper_trace.log

log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n


> Session expire/close flooding renders heartbeats to delay significantly
> -----------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-1049
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1049
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.3.2
>         Environment: CentOS 5.3, three node ZK ensemble
>            Reporter: Chang Song
>            Priority: Critical
>         Attachments: zk_ping_latency.pdf
>
>
> Let's say we have 100 clients (group A) already connected to three-node ZK ensemble with
session timeout of 15 second.  And we have 1000 clients (group B) already connected to the
same ZK ensemble, all watching several nodes (with 15 second session timeout)
> Consider a case in which All clients in group B suddenly hung or deadlocked (JVM OOME)
all at the same time. 15 seconds later, all sessions in group B gets expired, creating session
closing stampede. Depending on the number of this clients in group B, all request/response
ZK ensemble should process get delayed up to 8 seconds (1000 clients we have tested).
> This delay causes some clients in group A their sessions expired due to delay in getting
heartbeat response. This causes normal servers to drop out of clusters. This is a serious
problem in our installation, since some of our services running batch servers or CI servers
creating the same scenario as above almost everyday.
> I am attaching a graph showing ping response time delay.
> I think ordering of creating/closing sessions and ping exchange isn't important (quorum
state machine). at least ping request / response should be handle independently (different
queue and different thread) to keep realtime-ness of ping.
> As a workaround, we are raising session timeout to 50 seconds.
> But this causes max. failover of cluster to significantly increased, thus initial QoS
we promised cannot be met.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message