ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley" <jhur...@hortonworks.com>
Subject Re: Review Request 38933: JPA Threads Stuck In ConcurrencyManager Freezes Ambari
Date Thu, 01 Oct 2015 18:20:32 GMT


> On Oct. 1, 2015, 2:10 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
line 451
> > <https://reviews.apache.org/r/38933/diff/1/?file=1088667#file1088667line451>
> >
> >     Minor: For consistency
> >      
> >      protected static final String EXPERIMENTAL_CONCURRENCY_STAGE_PROCESSING_ENABLED=
"experimental.concurrency.stage_processing.enabled";

Renamed.


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38933/#review101253
-----------------------------------------------------------


On Oct. 1, 2015, 2:02 p.m., Jonathan Hurley wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38933/
> -----------------------------------------------------------
> 
> (Updated Oct. 1, 2015, 2:02 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Mahadev Konar, Nate Cole, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-13291
>     https://issues.apache.org/jira/browse/AMBARI-13291
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Related to AMBARI-13245, Ambari can stop functioning with dead threads during an request
that's currently in progress. The cause of this problem appears to be a performance fix which
processes {{StageEntity}} instances asynchronously to construct {{Stage}} instances.
> 
> Consider:
> - We have _n_ {{StageEntity}} instances from the database, each with a reference to the
single {{EntityManager}} instance that retrieved it from the DAO
> - Now, we use {{Parallel}} and spawn a bunch of threads to construct {{Stage}} instances
from the {{StageEnitty}}
> - However, {{StageEntity}} has {{LAZY}} associations which are accessed in the construction
of {{Stage}} - however, those associations are retrieved with the _same_ {{EntityManager}}
instance.
> - Becuase we have multiple threads constructing {{Stage}}s, this means we have *concurrent
access to the same EntityManager which is a no-no*.
> 
> It was previously thought that only a single threadpool in {{Parallel}} would prevent
this issue. However, it occurred again during a stack distribution of a running cluster:
> 
> {code}
> Exception Description: A signal was attempted before wait() on ConcurrencyManager. This
normally means that an attempt was made to
> commit or rollback a transaction before it was started, or to rollback a transaction
twice.
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> 	at org.apache.ambari.server.utils.Parallel.forLoop(Parallel.java:214)
> 	at org.apache.ambari.server.utils.Parallel.forLoop(Parallel.java:128)
> 	at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:215)
> 	at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:230)
> 	at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:195)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: Exception [EclipseLink-2004] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.ConcurrencyException
> Exception Description: A signal was attempted before wait() on ConcurrencyManager. This
normally means that an attempt was made to
> commit or rollback a transaction before it was started, or to rollback a transaction
twice.
> 	at org.eclipse.persistence.exceptions.ConcurrencyException.signalAttemptedBeforeWait(ConcurrencyException.java:84)
> 	at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseReadLock(ConcurrencyManager.java:468)
> 	at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseReadLock(CacheKey.java:468)
> 	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1044)
> 	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:955)
> 	at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
> 	at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
> 	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3942)
> 	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3894)
> 	at org.eclipse.persistence.mappings.CollectionMapping.buildElementUnitOfWorkClone(CollectionMapping.java:308)
> 	at org.eclipse.persistence.mappings.CollectionMapping.buildElementClone(CollectionMapping.java:321)
> 	at org.eclipse.persistence.internal.queries.ContainerPolicy.addNextValueFromIteratorInto(ContainerPolicy.java:217)
> 	at org.eclipse.persistence.mappings.CollectionMapping.buildCloneForPartObject(CollectionMapping.java:223)
> 	at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:60)
> 	at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
> 	at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
> 	at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
> 	at org.eclipse.persistence.indirection.IndirectList.buildDelegate(IndirectList.java:252)
> 	at org.eclipse.persistence.indirection.IndirectList.getDelegate(IndirectList.java:423)
> 	at org.eclipse.persistence.indirection.IndirectList$1.<init>(IndirectList.java:551)
> 	at org.eclipse.persistence.indirection.IndirectList.listIterator(IndirectList.java:550)
> 	at org.eclipse.persistence.indirection.IndirectList.iterator(IndirectList.java:514)
> 	at org.apache.ambari.server.actionmanager.Stage.<init>(Stage.java:157)
> 	at org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:77)
> 	at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl$1.run(ActionDBAccessorImpl.java:218)
> 	at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl$1.run(ActionDBAccessorImpl.java:215)
> 	at org.apache.ambari.server.utils.Parallel$1.call(Parallel.java:178)
> 	at org.apache.ambari.server.utils.Parallel$1.call(Parallel.java:173)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> {code}
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/annotations/Experimental.java PRE-CREATION

>   ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
0f439de 
>   ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
67dcfa5 
>   ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
074fbb4 
> 
> Diff: https://reviews.apache.org/r/38933/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message