Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5EF42200BA5 for ; Wed, 19 Oct 2016 18:51:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5DAF4160ADC; Wed, 19 Oct 2016 16:51:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 711CB160AEA for ; Wed, 19 Oct 2016 18:51:05 +0200 (CEST) Received: (qmail 30945 invoked by uid 500); 19 Oct 2016 16:50:59 -0000 Mailing-List: contact issues-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list issues@ambari.apache.org Received: (qmail 30888 invoked by uid 99); 19 Oct 2016 16:50:59 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Oct 2016 16:50:59 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 2C7AE2C4C84 for ; Wed, 19 Oct 2016 16:50:59 +0000 (UTC) Date: Wed, 19 Oct 2016 16:50:59 +0000 (UTC) From: "Hudson (JIRA)" To: issues@ambari.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (AMBARI-18456) Refactor Unnecessary In-Memory Locks Around Business Objects MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 19 Oct 2016 16:51:06 -0000 [ https://issues.apache.org/jira/browse/AMBARI-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589226#comment-15589226 ] Hudson commented on AMBARI-18456: --------------------------------- FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #5826 (See [https://builds.apache.org/job/Ambari-trunk-Commit/5826/]) AMBARI-18456 - Refactor Unnecessary In-Memory Locks Around Business (jhurley: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=d6a847154e80158a0be0a26ee7dfc0d6dccac686]) * (edit) ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponent.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/events/EventsTest.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/orm/OrmTestHelper.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ConcurrentServiceConfigVersionTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClustersDeadlockTest.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatMonitor.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClustersTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeActionTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ComponentVersionCheckActionTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterImplTest.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ServiceComponentHostConcurrentWriteDeadlockTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/configuration/RecoveryConfigHelperTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterDeadlockTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/ServiceComponentTest.java * (edit) ambari-server/src/main/java/org/apache/ambari/annotations/ExperimentalFeature.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ComponentResourceProvider.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/agent/HeartbeatProcessorTest.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/state/ServiceTest.java > Refactor Unnecessary In-Memory Locks Around Business Objects > ------------------------------------------------------------ > > Key: AMBARI-18456 > URL: https://issues.apache.org/jira/browse/AMBARI-18456 > Project: Ambari > Issue Type: Epic > Components: ambari-server > Affects Versions: 2.5.0 > Reporter: Jonathan Hurley > Assignee: Jonathan Hurley > Labels: branch-feature-AMBARI-18456 > Fix For: 2.5.0 > > > The top 4 business objects in Ambari: > - ClusterImpl > - ServiceImpl > - ServiceComponentImpl > - ServiceComponentHostImpl > All use {{ReadWriteLock}} implementations to prevent dirty reads and concurrent writes. However, {{ClusterImpl}} exposes a "global" {{ReadWriteLock}} which the other business objects share. This causes tremendous problems with deadlocks, especially on slow databases. > Consider the case where you have 3 threads: > # thread-1 acquires {{ClusterReadLock}} > # thread-2 acquires {{ServiceComponenWriteLock}} > # thread-3 tries to get {{ClusterWriteLock}} and is blocked by {{thread-1}} > # thread-2 tries to get {{ClusterReadLock}} and is blocked by {{thread-3}} > # thread-1 tries to get {{ServiceComponentReadLock}} and is blocked by {{thread-2}} > Essentially, the exposure of the "cluster global lock" causes problems since multiple threads can acquire other internal locks and be blocked waiting on the global lock. > In general, I don't believe that the read locks help at all. Ambari usually encounters these locks while try to display web page information. Once displayed, the locks are removed and the information is already stale if there were write threads waiting. > These locks should be investigated and, for the most part, except in some cases involving concurrent writes, removed. > Part of the problem revolves around our assumption about how the ReadWriteLock works. The issue in the above scenario is that the clusterWriteLock request is pending. This actually blocks all subsequent readers even though the lock is not fair. > FYI, this code shows that a reader, in unfair mode, will wait when there is a waiting writer: > {noformat:title=Output} > Waiting for a read lock... > Read lock acquired! > Waiting for a write lock... > Trying to acquire a second read lock... > {noformat} > {code} > import java.util.concurrent.locks.ReentrantReadWriteLock; > public class Test { > private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock(false); > public static void main(String[] args) throws InterruptedException { > // A reader which takes too long to finish > new Thread() { > @Override > public void run() { > System.out.println("Waiting for a read lock..."); > lock.readLock().lock(); > System.out.println("Read lock acquired!"); > try { > try { > Thread.sleep(1000 * 60 * 60); > } catch (InterruptedException e) { > } > } finally { > lock.readLock().unlock(); > } > } > }.start(); > Thread.sleep(3000); > // A writer which will be waiting > new Thread() { > @Override > public void run() { > System.out.println("Waiting for a write lock..."); > lock.writeLock().lock(); > System.out.println("Write lock acquired!"); > lock.writeLock().unlock(); > } > }.start(); > Thread.sleep(3000); > // Another reader > new Thread() { > @Override > public void run() { > System.out.println("Trying to acquire a second read lock..."); > lock.readLock().lock(); > try { > System.out.println("Second read lock acquired successfully!!"); > } finally { > lock.readLock().unlock(); > } > } > }.start(); > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)