geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dinesh Akhand <dines...@amdocs.com>
Subject RE: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
Date Thu, 01 Nov 2018 11:34:01 GMT

In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a
send of a primary put from another server. As part of that put operation, its delivering the
event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated.
This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the
cache initialization is complete.

Regarding above command,  First time this happed when JVM is getting up and Function start
executing on that same JVM , but JVM is not initialized till now.
So avoiding such issue, I am trying to add the check with new method is geode.

Which tells user function that IsJVMinitializationCompleted then only function execute should
work.

Thanks,
Dinesh Akhand

-----Original Message-----
From: Barry Oglesby <boglesby@pivotal.io> 
Sent: Thursday, October 25, 2018 11:09 PM
To: user@geode.apache.org
Cc: dev@geode.apache.org
Subject: Re: Thread block on org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)

The main thread has locked the CacheFactory (0x000000021de38d70) and is initializing the cache.
At this time it is registering a partitioned region by getting a distributed lock. It is waiting
for a reply to this request from a remote member:

"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on condition [0x00007f8563ca2000]
   java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000001c9d4b2e8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858)
at
org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242)
at
org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537)
at
org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683)
at
org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626)
at
org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883)
at
org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318)
at
org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987)
- locked <0x000000021dd35258> (a
org.apache.geode.internal.cache.PartitionedRegion)
at
org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157)
at
org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004)
at
org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558)
at
org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512)
at
org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338)
at
org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390)
at
org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758)
- locked <0x000000021de39cf0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212)
- locked <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52)
at
org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860)
at
org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772)
at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699)
at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231)

In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a
send of a primary put from another server. As part of that put operation, its delivering the
event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated.
This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the
cache initialization is complete.

"P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024
shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0
tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry [0x00007f84d0f9a000]
   java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282)
- waiting to lock <0x000000021de38d70> (a java.lang.Class for
org.apache.geode.cache.CacheFactory)
at
amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43)
at
amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35)
at
amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19)
at
amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27)
at
amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
- locked <0x000000021ed39f00> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236)
at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98)
at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110)
at
amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255)
at
amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083)
at
amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103)
at
amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798)
at amdocs.imdg.model.Customer.fromData(Customer.java:696)
- locked <0x00000002469dd138> (a amdocs.imdg.model.Customer) at
org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372)
at
org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395)
at
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)
at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)
at
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)
at
org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)
at
org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)
- locked <0x000000021e7b50c8> (a
org.apache.geode.internal.cache.VMCachedDeserializable)
at
org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975)
at
org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown
Source)
at
org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007)
at
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970)
at
amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971)
at
org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928)
at
org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786)
at
org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689)
at java.lang.Thread.run(Thread.java:745)

Then, there are 9 more P2P message reader threads waiting for the GenericKeyedObjectPool (0x000000021ed39f00)
held by this thread.

I'm not sure whether or not this is a deadlock. It would be interesting to see additional
thread dumps for this server to see if the main thread is processing or stuck and also thread
dumps for the other server(s) to see specifically what the thread processing the DLockRequestMessage
is doing (thats the remote thread that the main thread above is waiting for).

Would it be possible to create the VSDCountersManager in the main thread before the P2P message
reader thread attempts to do it (like in a Function init)?

Thanks,
Barry Oglesby



On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <abaker@pivotal.io> wrote:

> Sorry I don’t think I understand your question.  Did you encounter a 
> problem with getAnyInstance()?
>
> Anthony
>
>
> > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <dineshak@amdocs.com> wrote:
> >
> > No dead local was there , reader thread are waiting mostly.
> > PFA whole stack strace.
> >
> > Thanks,
> > Dinesh Akhand
>
>
“Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any
emails sent to Amdocs will be processed and stored using such system and are accessible by
third party providers of such system on a limited basis. Your sending of emails to Amdocs
evidences your consent to the use of such system and such processing, storing and access”.
Mime
View raw message