zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (ZOOKEEPER-2684) Fix a crashing bug in the mixed workloads commit processor
Date Tue, 14 Feb 2017 18:20:41 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866318#comment-15866318
] 

Ryan Zhang edited comment on ZOOKEEPER-2684 at 2/14/17 6:20 PM:
----------------------------------------------------------------

Unfortunately, this is only half of the story. I tested the build and see the following crash

atla-cbo-26-sr4.prod.twttr.net:  2017-02-14 17:55:42,323 - ERROR [CommitProcessor:4] - Severe
unrecoverable error, from thread : CommitProcessor:4
atla-cbo-26-sr4.prod.twttr.net:  org.apache.zookeeper.KeeperException$RuntimeInconsistencyException:
KeeperErrorCode = RuntimeInconsistency
atla-cbo-26-sr4.prod.twttr.net:  	at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276)
atla-cbo-26-sr4.prod.twttr.net:  2017-02-14 18:12:25,515 - ERROR [CommitProcessor:4] - Got
request sessionid:0x40069edcb5b7b7f type:create cxid:0x36 zxid:0x38205009d26 txntype:1 reqpath:n/a
atla-cbo-26-sr4.prod.twttr.net:   but we are expecting request sessionid:0x40069edcb5b7b7f
type:delete cxid:0x30 zxid:0xfffffffffffffffe txntype:unknown reqpath:/loadgen/load_1759491424
atla-cbo-26-sr4.prod.twttr.net:  


Stay tuned, I am adding back the verbose logs and will report on what I found


was (Author: nerdyyatrice):
Unfortunately, this is only half of the story. I tested the build and see the following crash

atla-avu-37-sr1.prod.twttr.net:  2017-02-14 15:04:06,180 - ERROR [CommitProcessor:5] - Severe
unrecoverable error, from thread : CommitProcessor:5
atla-avu-37-sr1.prod.twttr.net:  org.apache.zookeeper.KeeperException$RuntimeInconsistencyException:
KeeperErrorCode = RuntimeInconsistency
atla-avu-37-sr1.prod.twttr.net:  	at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:276)

Stay tuned, I am adding back the verbose logs and will report on what I found

> Fix a crashing bug in the mixed workloads commit processor
> ----------------------------------------------------------
>
>                 Key: ZOOKEEPER-2684
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2684
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.6.0
>         Environment: with pretty heavy load on a real cluster
>            Reporter: Ryan Zhang
>            Assignee: Ryan Zhang
>            Priority: Blocker
>         Attachments: ZOOKEEPER-2684.patch
>
>
> We deployed our build with ZOOKEEPER-2024 and it quickly started to crash with the following
error
> atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:24:42,305 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268)
– Got cxid 0x119fa expected 0x11fc5 for client session id 1009079ba470055
> atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:32:04,746 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268)
– Got cxid 0x698 expected 0x928 for client session id 4002eeb3fd0009d
> atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:34:46,648 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268)
– Got cxid 0x8904 expected 0x8f34 for client session id 51b8905c90251
> atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:43:46,834 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268)
– Got cxid 0x3a8d expected 0x3ebc for client session id 2051af11af900cc
> clearly something is not right in the new commit processor per session queue implementation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message