From issues-return-871-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Wed Sep 4 02:20:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3B555180637 for ; Wed, 4 Sep 2019 04:20:02 +0200 (CEST) Received: (qmail 32258 invoked by uid 500); 4 Sep 2019 05:57:38 -0000 Mailing-List: contact issues-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zookeeper.apache.org Delivered-To: mailing list issues@zookeeper.apache.org Received: (qmail 32247 invoked by uid 99); 4 Sep 2019 05:57:38 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Sep 2019 05:57:38 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id BB5E3E305B for ; Wed, 4 Sep 2019 02:20:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 279617806CB for ; Wed, 4 Sep 2019 02:20:00 +0000 (UTC) Date: Wed, 4 Sep 2019 02:20:00 +0000 (UTC) From: "ASF GitHub Bot (Jira)" To: issues@zookeeper.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (ZOOKEEPER-3531) Synchronization on ACLCache cause cluster to hang when network/disk issues happen during datatree serialization MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ZOOKEEPER-3531?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ZOOKEEPER-3531: -------------------------------------- Labels: pull-request-available (was: ) > Synchronization on ACLCache cause cluster to hang when network/disk issue= s happen during datatree serialization > -------------------------------------------------------------------------= -------------------------------------- > > Key: ZOOKEEPER-3531 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3531 > Project: ZooKeeper > Issue Type: Bug > Affects Versions: 3.5.2, 3.5.3, 3.5.4, 3.5.5 > Reporter: Chang Lou > Priority: Critical > Labels: pull-request-available > Attachments: fix.patch, generator.py > > > During our ZooKeeper fault injection testing, we observed that sometimes = the ZK cluster could hang (requests time out, node status shows ok). After = inspecting the issue, we believe this is caused by I/O (serializing ACLCach= e) inside a critical section. The bug is essentially similar to what is des= cribed in ZooKeeper-2201. > org.apache.zookeeper.server.DataTree#serialize calls the aclCache.seriali= ze when doing dataree serialization, however, org.apache.zookeeper.server.R= eferenceCountedACLCache#serialize could get stuck at OutputArchieve.writeIn= t due to potential network/disk issues. This can cause the system experienc= es hanging issues similar to ZooKeeper-2201 (any attempt to create/delete/m= odify the DataNode will cause the leader to hang at the beginning of the re= quest processor chain). The root cause is the lock contention between: > * org.apache.zookeeper.server.DataTree#serialize -> org.apache.zookeeper= .server.ReferenceCountedACLCache#serialize=C2=A0 > * PrepRequestProcessor#getRecordForPath -> org.apache.zookeeper.server.D= ataTree#getACL(org.apache.zookeeper.server.DataNode) -> org.apache.zookeepe= r.server.ReferenceCountedACLCache#convertLong > When the snapshot gets stuck in acl serialization, it would prevent all o= ther operations to ReferenceCountedACLCache. Since getRecordForPath calls R= eferenceCountedACLCache#convertLong, any op triggering getRecordForPath wil= l cause the leader to hang at the beginning of the request processor chain: > {code:java} > org.apache.zookeeper.server.ReferenceCountedACLCache.convertLong(Referenc= eCountedACLCache.java:87) > org.apache.zookeeper.server.DataTree.getACL(DataTree.java:734) > =C2=A0=C2=A0=C2=A0- locked org.apache.zookeeper.server.DataNode@4a062b7d > org.apache.zookeeper.server.ZKDatabase.aclForNode(ZKDatabase.java:371) > org.apache.zookeeper.server.PrepRequestProcessor.getRecordForPath(PrepReq= uestProcessor.java:170) > =C2=A0=C2=A0=C2=A0- locked java.util.ArrayDeque@3f7394f7 > org.apache.zookeeper.server.PrepRequestProcessor.pRequest2Txn(PrepRequest= Processor.java:417) > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProc= essor.java:757) > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor= .java:145) > {code} > Similar to ZooKeeper-2201, the leader can still send out heartbeats so th= e cluster will not recover until the network/disk issue resolves.=C2=A0=C2= =A0 > Steps to reproduce this bug: > # start a cluster with 1 leader and n followers > # manually create some ACLs, to enlarge the window of dumping acls so it= would be more likely to hang at serializing ACLCache when delay happens. (= we wrote a script to generate such workloads, see attachments) > # inject long network/disk write delays and run some benchmarks to trigg= er snapshots > # once stuck, you should observe new requests to the cluster would fail. > Essentially the core problem is the OutputArchive write should not be kep= t inside this synchronization block. So a straightforward solution is to mo= ve writes out of sync block: do a copy inside the sync block and perform vu= lnerable network writes afterwards. The patch for this solution is attached= and verified.=C2=A0 Another more systematic fix is perhaps replacing all s= ynchronized methods in the ReferenceCountedACLCache with ConcurrentHashMap.= =C2=A0 > We double checked that the issue remains in the latest version of master = branch (68c21988d55c57e483370d3ee223c22da2d1bbcf).=C2=A0 > Attachments are 1) patch for fix and regression test 2) scripts to genera= te workloads to fill ACL cache -- This message was sent by Atlassian Jira (v8.3.2#803003)