geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Barry Oglesby <bogle...@pivotal.io>
Subject Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Date Thu, 25 Oct 2018 17:39:02 GMT
The main thread has locked the CacheFactory (0x000000021de38d70) and is
initializing the cache. At this time it is registering a partitioned region
by getting a distributed lock. It is waiting for a reply to this request
from a remote member:

"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on
condition [0x00007f8563ca2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000001c9d4b2e8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858)
at
org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242)
at
org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537)
at
org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626)
at
org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883)
at
org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318)
at
org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987)
- locked <0x000000021dd35258> (a
org.apache.geode.internal.cache.PartitionedRegion)
at
org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157)
at
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004)
at
org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512)
at
org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
at
org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758)
- locked <0x000000021de39cf0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
at
org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860)
at
org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772)
at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699)
at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231)

In the meantime, a PutMessage is being processed by a P2P message reader
thread. This is a send of a primary put from another server. As part of
that put operation, its delivering the event to a gateway sender which
ultimately causes a VSDCountersManager to be instantiated. This is blocked
waiting for the CacheFactory. The CacheFactory won't be available until the
cache initialization is complete.

"P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024
shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0
tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry
[0x00007f84d0f9a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at
amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at
amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at
amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at
amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ed39f00> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at
amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at
amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at
amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at
amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x00000002469dd138> (a amdocs.imdg.model.Customer)
at
org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at
org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)
at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)
at
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)
at
org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)
- locked <0x000000021e7b50c8> (a
org.apache.geode.internal.cache.VMCachedDeserializable)
at
org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975)
at
org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown
Source)
at
org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007)
at
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970)
at
amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971)
at
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928)
at
org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786)
at
org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689)
at java.lang.Thread.run(Thread.java:745)

Then, there are 9 more P2P message reader threads waiting for the
GenericKeyedObjectPool (0x000000021ed39f00) held by this thread.

I'm not sure whether or not this is a deadlock. It would be interesting to
see additional thread dumps for this server to see if the main thread is
processing or stuck and also thread dumps for the other server(s) to see
specifically what the thread processing the DLockRequestMessage is doing
(thats the remote thread that the main thread above is waiting for).

Would it be possible to create the VSDCountersManager in the main thread
before the P2P message reader thread attempts to do it (like in a Function
init)?

Thanks,
Barry Oglesby



On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <abaker@pivotal.io> wrote:

> Sorry I don’t think I understand your question.  Did you encounter a
> problem with getAnyInstance()?
>
> Anthony
>
>
> > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <dineshak@amdocs.com> wrote:
> >
> > No dead local was there , reader thread are waiting mostly.
> > PFA whole stack strace.
> >
> > Thanks,
> > Dinesh Akhand
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message