Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DA29B101EB for ; Fri, 5 Jun 2015 08:30:00 +0000 (UTC) Received: (qmail 74213 invoked by uid 500); 5 Jun 2015 08:30:00 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 74146 invoked by uid 500); 5 Jun 2015 08:30:00 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 73800 invoked by uid 99); 5 Jun 2015 08:30:00 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jun 2015 08:30:00 +0000 Date: Fri, 5 Jun 2015 08:30:00 +0000 (UTC) From: "Benoit Perroud (JIRA)" To: dev@ambari.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AMBARI-11713) Deleting a service does not remove alert_definition, leading to a deadlock when re-adding the service MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMBARI-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Perroud updated AMBARI-11713: ------------------------------------ Description: When removing a service (via the API, for instance {{curl -u $ambariuser -i -H 'X-Requested-By: ambari' -X DELETE http://ambari-server:8080/api/v1/clusters/CLUSTER01/services/AMBARI_METRICS}}), the entries in {{alert_definition}} table are not removed. When re-adding the service, the {{uni_alert_def_name}} constraint fails, leading to a deadlock (looks like the lock is not released properly). Constraint exception: {code} Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uni_alert_def_name" Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists. Error Code: 0 Call: INSERT INTO alert_definition (definition_id, cluster_id, component_name, definition_name, description, ENABLED, HASH, ignore_host, label, schedule_interval, scope, service_name, alert_source, source_type) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) bind => [14 parameters bound] at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:898) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631) at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149) at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1836) at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4244) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5594) at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:453) at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863) at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963) at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207) at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517) at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400) at org.apache.ambari.server.orm.dao.DaoUtils.selectSingle(DaoUtils.java:71) at org.apache.ambari.server.orm.dao.AlertDispatchDAO.findDefaultServiceGroup(AlertDispatchDAO.java:346) at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53) at org.apache.ambari.server.orm.dao.AlertDefinitionDAO.create(AlertDefinitionDAO.java:307) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68) at org.apache.ambari.server.events.listeners.alerts.AlertServiceStateListener.onAmbariEvent(AlertServiceStateListener.java:148) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74) at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314) at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34) at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uni_alert_def_name" Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) {code} Stacktrace {code} "qtp-client-24" prio=10 tid=0x00007fdc656e4800 nid=0xe79 waiting on condition [0x00007fdc266bb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000079e3364f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945) at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persistConfigMapping(ConfigGroupImpl.java:419) at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.saveIfPersisted(ConfigGroupImpl.java:456) at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persist(ConfigGroupImpl.java:323) at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.updateConfigGroups(ConfigGroupResourceProvider.java:600) - locked <0x00000007a0067b30> (a org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider) at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.access$300(ConfigGroupResourceProvider.java:58) at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider$4.invoke(ConfigGroupResourceProvider.java:269) {code} was: When removing a service (via the API, for instance {{curl -u $ambariuser -i -H 'X-Requested-By: ambari' -X DELETE http://daplab-wn-06:8080/api/v1/clusters/CLUSTER01/services/AMBARI_METRICS}}), the entries in {{alert_definition}} table are not removed. When re-adding the service, the {{uni_alert_def_name}} constraint fails, leading to a deadlock (looks like the lock is not released properly). Constraint exception: {code} Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uni_alert_def_name" Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists. Error Code: 0 Call: INSERT INTO alert_definition (definition_id, cluster_id, component_name, definition_name, description, ENABLED, HASH, ignore_host, label, schedule_interval, scope, service_name, alert_source, source_type) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) bind => [14 parameters bound] at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:898) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631) at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149) at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1836) at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4244) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5594) at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:453) at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863) at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963) at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207) at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517) at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400) at org.apache.ambari.server.orm.dao.DaoUtils.selectSingle(DaoUtils.java:71) at org.apache.ambari.server.orm.dao.AlertDispatchDAO.findDefaultServiceGroup(AlertDispatchDAO.java:346) at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53) at org.apache.ambari.server.orm.dao.AlertDefinitionDAO.create(AlertDefinitionDAO.java:307) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68) at org.apache.ambari.server.events.listeners.alerts.AlertServiceStateListener.onAmbariEvent(AlertServiceStateListener.java:148) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74) at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314) at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34) at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uni_alert_def_name" Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) {code} Stacktrace {code} "qtp-client-24" prio=10 tid=0x00007fdc656e4800 nid=0xe79 waiting on condition [0x00007fdc266bb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000079e3364f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945) at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persistConfigMapping(ConfigGroupImpl.java:419) at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.saveIfPersisted(ConfigGroupImpl.java:456) at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persist(ConfigGroupImpl.java:323) at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.updateConfigGroups(ConfigGroupResourceProvider.java:600) - locked <0x00000007a0067b30> (a org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider) at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.access$300(ConfigGroupResourceProvider.java:58) at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider$4.invoke(ConfigGroupResourceProvider.java:269) {code} > Deleting a service does not remove alert_definition, leading to a deadlock when re-adding the service > ----------------------------------------------------------------------------------------------------- > > Key: AMBARI-11713 > URL: https://issues.apache.org/jira/browse/AMBARI-11713 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Affects Versions: 2.0.1 > Reporter: Benoit Perroud > Priority: Minor > > When removing a service (via the API, for instance {{curl -u $ambariuser -i -H 'X-Requested-By: ambari' -X DELETE http://ambari-server:8080/api/v1/clusters/CLUSTER01/services/AMBARI_METRICS}}), the entries in {{alert_definition}} table are not removed. > When re-adding the service, the {{uni_alert_def_name}} constraint fails, leading to a deadlock (looks like the lock is not released properly). > Constraint exception: > {code} > Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException > Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uni_alert_def_name" > Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists. > Error Code: 0 > Call: INSERT INTO alert_definition (definition_id, cluster_id, component_name, definition_name, description, ENABLED, HASH, ignore_host, label, schedule_interval, scope, service_name, alert_source, source_type) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) > bind => [14 parameters bound] > at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) > at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611) > at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:898) > at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962) > at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631) > at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149) > at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134) > at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1836) > at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4244) > at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5594) > at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:453) > at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863) > at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963) > at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207) > at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517) > at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400) > at org.apache.ambari.server.orm.dao.DaoUtils.selectSingle(DaoUtils.java:71) > at org.apache.ambari.server.orm.dao.AlertDispatchDAO.findDefaultServiceGroup(AlertDispatchDAO.java:346) > at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53) > at org.apache.ambari.server.orm.dao.AlertDefinitionDAO.create(AlertDefinitionDAO.java:307) > at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68) > at org.apache.ambari.server.events.listeners.alerts.AlertServiceStateListener.onAmbariEvent(AlertServiceStateListener.java:148) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74) > at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314) > at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34) > at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:100) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uni_alert_def_name" > Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists. > at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) > at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) > at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) > {code} > > Stacktrace > {code} > "qtp-client-24" prio=10 tid=0x00007fdc656e4800 nid=0xe79 waiting on condition [0x00007fdc266bb000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000000079e3364f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197) > at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945) > at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persistConfigMapping(ConfigGroupImpl.java:419) > at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.saveIfPersisted(ConfigGroupImpl.java:456) > at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persist(ConfigGroupImpl.java:323) > at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.updateConfigGroups(ConfigGroupResourceProvider.java:600) > - locked <0x00000007a0067b30> (a org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider) > at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.access$300(ConfigGroupResourceProvider.java:58) > at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider$4.invoke(ConfigGroupResourceProvider.java:269) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)