zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chang lou (Jira)" <j...@apache.org>
Subject [jira] [Created] (ZOOKEEPER-3531) Synchronization on ACLCache cause cluster to hang when network/disk issues happen during datatree serialization
Date Mon, 02 Sep 2019 21:03:00 GMT
chang lou created ZOOKEEPER-3531:

             Summary: Synchronization on ACLCache cause cluster to hang when network/disk
issues happen during datatree serialization
                 Key: ZOOKEEPER-3531
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3531
             Project: ZooKeeper
          Issue Type: Bug
    Affects Versions: 3.5.5, 3.5.4, 3.5.3, 3.5.2
            Reporter: chang lou
         Attachments: fix.patch, generator.py

During our ZooKeeper fault injection testing, we observed that sometimes the ZK cluster could
hang (requests time out, node status shows ok). After inspecting the issue, we believe this
is caused by I/O (serializing ACLCache) inside a critical section. The bug is essentially
similar to what is described in ZooKeeper-2201.


org.apache.zookeeper.server.DataTree#serialize calls the aclCache.serialize when doing dataree
serialization, however, org.apache.zookeeper.server.ReferenceCountedACLCache#serialize could
get stuck at OutputArchieve.writeInt due to potential network/disk issues. This can cause
the system experiences hanging issues similar to ZooKeeper-2201 (any attempt to create/delete/modify
the DataNode will cause the leader to hang at the beginning of the request processor chain).
The root cause is the lock contention between:

 * org.apache.zookeeper.server.DataTree#serialize -> org.apache.zookeeper.server.ReferenceCountedACLCache#serialize 
 * PrepRequestProcessor#getRecordForPath -> org.apache.zookeeper.server.DataTree#getACL(org.apache.zookeeper.server.DataNode)
-> org.apache.zookeeper.server.ReferenceCountedACLCache#convertLong

When the snapshot gets stuck in acl serialization, it would prevent all other operations to
ReferenceCountedACLCache. Since getRecordForPath calls ReferenceCountedACLCache#convertLong,
any op triggering getRecordForPath will cause the leader to hang at the beginning of the request
processor chain:


   - locked org.apache.zookeeper.server.DataNode@4a062b7d
   - locked java.util.ArrayDeque@3f7394f7

SImilar to ZooKeeper-2201, the leader can still send out heartbeats so the cluster will not
recover until the network/disk issue resolves.  


Steps to reproduce this bug:
 # start a cluster with 1 leader and n followers
 # manually create some ACLs, to enlarge the window of dumping acls so it would be more likely
to hang at serializing ACLCache when delay happens. (we wrote a script to generate such workloads,
see attachments)
 # inject long network/disk write delays and run some benchmarks to trigger snapshots
 # once stuck, you should observe new requests to the cluster would fail.


Essentially the core problem is the OutputArchive write should not be kept inside this synchronization
block. So a straightforward solution is to move writes out of sync block: do a copy inside
the sync block and perform vulnerable network writes afterwards. The patch for this solution
is attached and verified.  Another more systematic fix is perhaps replacing all synchronized
methods in the ReferenceCountedACLCache with ConcurrentHashMap. 


We double checked that the issue remains in the latest version of master branch (68c21988d55c57e483370d3ee223c22da2d1bbcf). 


Attachments are 1) patch for fix and regression test 2) scripts to generate workloads to fill
ACL cache

This message was sent by Atlassian Jira

View raw message