ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benoit Perroud (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-11713) Deleting a service does not remove alert_definition, leading to a deadlock when re-adding the service
Date Fri, 05 Jun 2015 08:30:00 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Benoit Perroud updated AMBARI-11713:
------------------------------------
    Description: 
When removing a service (via the API, for instance {{curl -u $ambariuser -i -H 'X-Requested-By:
ambari' -X DELETE http://ambari-server:8080/api/v1/clusters/CLUSTER01/services/AMBARI_METRICS}}),
the entries in {{alert_definition}} table are not removed.

When re-adding the service, the {{uni_alert_def_name}} constraint fails, leading to a deadlock
(looks like the lock is not released properly).

Constraint exception:
{code}
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates
unique constraint "uni_alert_def_name"
  Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists.
Error Code: 0
Call: INSERT INTO alert_definition (definition_id, cluster_id, component_name, definition_name,
description, ENABLED, HASH, ignore_host, label, schedule_interval, scope, service_name, alert_source,
source_type) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        bind => [14 parameters bound]
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:898)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1836)
        at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4244)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5594)
        at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:453)
        at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863)
        at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963)
        at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
        at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517)
        at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400)
        at org.apache.ambari.server.orm.dao.DaoUtils.selectSingle(DaoUtils.java:71)
        at org.apache.ambari.server.orm.dao.AlertDispatchDAO.findDefaultServiceGroup(AlertDispatchDAO.java:346)
        at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53)
        at org.apache.ambari.server.orm.dao.AlertDefinitionDAO.create(AlertDefinitionDAO.java:307)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)
        at org.apache.ambari.server.events.listeners.alerts.AlertServiceStateListener.onAmbariEvent(AlertServiceStateListener.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74)
        at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314)
        at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
        at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint
"uni_alert_def_name"
  Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
{code}

 
Stacktrace
{code}
"qtp-client-24" prio=10 tid=0x00007fdc656e4800 nid=0xe79 waiting on condition [0x00007fdc266bb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000079e3364f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
        at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persistConfigMapping(ConfigGroupImpl.java:419)
        at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.saveIfPersisted(ConfigGroupImpl.java:456)
        at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persist(ConfigGroupImpl.java:323)
        at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.updateConfigGroups(ConfigGroupResourceProvider.java:600)
        - locked <0x00000007a0067b30> (a org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider)
        at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.access$300(ConfigGroupResourceProvider.java:58)
        at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider$4.invoke(ConfigGroupResourceProvider.java:269)
{code}


  was:
When removing a service (via the API, for instance {{curl -u $ambariuser -i -H 'X-Requested-By:
ambari' -X DELETE http://daplab-wn-06:8080/api/v1/clusters/CLUSTER01/services/AMBARI_METRICS}}),
the entries in {{alert_definition}} table are not removed.

When re-adding the service, the {{uni_alert_def_name}} constraint fails, leading to a deadlock
(looks like the lock is not released properly).

Constraint exception:
{code}
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates
unique constraint "uni_alert_def_name"
  Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists.
Error Code: 0
Call: INSERT INTO alert_definition (definition_id, cluster_id, component_name, definition_name,
description, ENABLED, HASH, ignore_host, label, schedule_interval, scope, service_name, alert_source,
source_type) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        bind => [14 parameters bound]
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:898)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1836)
        at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4244)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5594)
        at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:453)
        at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863)
        at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963)
        at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
        at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517)
        at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400)
        at org.apache.ambari.server.orm.dao.DaoUtils.selectSingle(DaoUtils.java:71)
        at org.apache.ambari.server.orm.dao.AlertDispatchDAO.findDefaultServiceGroup(AlertDispatchDAO.java:346)
        at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53)
        at org.apache.ambari.server.orm.dao.AlertDefinitionDAO.create(AlertDefinitionDAO.java:307)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)
        at org.apache.ambari.server.events.listeners.alerts.AlertServiceStateListener.onAmbariEvent(AlertServiceStateListener.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74)
        at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314)
        at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
        at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint
"uni_alert_def_name"
  Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already exists.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
{code}

 
Stacktrace
{code}
"qtp-client-24" prio=10 tid=0x00007fdc656e4800 nid=0xe79 waiting on condition [0x00007fdc266bb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000079e3364f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
        at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persistConfigMapping(ConfigGroupImpl.java:419)
        at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.saveIfPersisted(ConfigGroupImpl.java:456)
        at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persist(ConfigGroupImpl.java:323)
        at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.updateConfigGroups(ConfigGroupResourceProvider.java:600)
        - locked <0x00000007a0067b30> (a org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider)
        at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.access$300(ConfigGroupResourceProvider.java:58)
        at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider$4.invoke(ConfigGroupResourceProvider.java:269)
{code}



> Deleting a service does not remove alert_definition, leading to a deadlock when re-adding
the service
> -----------------------------------------------------------------------------------------------------
>
>                 Key: AMBARI-11713
>                 URL: https://issues.apache.org/jira/browse/AMBARI-11713
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: 2.0.1
>            Reporter: Benoit Perroud
>            Priority: Minor
>
> When removing a service (via the API, for instance {{curl -u $ambariuser -i -H 'X-Requested-By:
ambari' -X DELETE http://ambari-server:8080/api/v1/clusters/CLUSTER01/services/AMBARI_METRICS}}),
the entries in {{alert_definition}} table are not removed.
> When re-adding the service, the {{uni_alert_def_name}} constraint fails, leading to a
deadlock (looks like the lock is not released properly).
> Constraint exception:
> {code}
> Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates
unique constraint "uni_alert_def_name"
>   Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already
exists.
> Error Code: 0
> Call: INSERT INTO alert_definition (definition_id, cluster_id, component_name, definition_name,
description, ENABLED, HASH, ignore_host, label, schedule_interval, scope, service_name, alert_source,
source_type) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
>         bind => [14 parameters bound]
>         at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
>         at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611)
>         at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:898)
>         at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:962)
>         at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:631)
>         at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
>         at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
>         at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1836)
>         at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4244)
>         at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5594)
>         at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:453)
>         at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863)
>         at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963)
>         at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
>         at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517)
>         at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400)
>         at org.apache.ambari.server.orm.dao.DaoUtils.selectSingle(DaoUtils.java:71)
>         at org.apache.ambari.server.orm.dao.AlertDispatchDAO.findDefaultServiceGroup(AlertDispatchDAO.java:346)
>         at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53)
>         at org.apache.ambari.server.orm.dao.AlertDefinitionDAO.create(AlertDefinitionDAO.java:307)
>         at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)
>         at org.apache.ambari.server.events.listeners.alerts.AlertServiceStateListener.onAmbariEvent(AlertServiceStateListener.java:148)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74)
>         at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314)
>         at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
>         at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:100)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique
constraint "uni_alert_def_name"
>   Detail: Key (cluster_id, definition_name)=(2, metrics_monitor_process_percent) already
exists.
>         at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
>         at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
>         at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
> {code}
>  
> Stacktrace
> {code}
> "qtp-client-24" prio=10 tid=0x00007fdc656e4800 nid=0xe79 waiting on condition [0x00007fdc266bb000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x000000079e3364f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
>         at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
>         at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persistConfigMapping(ConfigGroupImpl.java:419)
>         at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.saveIfPersisted(ConfigGroupImpl.java:456)
>         at org.apache.ambari.server.state.configgroup.ConfigGroupImpl.persist(ConfigGroupImpl.java:323)
>         at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.updateConfigGroups(ConfigGroupResourceProvider.java:600)
>         - locked <0x00000007a0067b30> (a org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider)
>         at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.access$300(ConfigGroupResourceProvider.java:58)
>         at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider$4.invoke(ConfigGroupResourceProvider.java:269)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message