ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alper Tekinalp <al...@evam.com>
Subject Re: Client got stucked on get operation
Date Thu, 06 Apr 2017 09:03:18 GMT
Hi Andrey.

You can find configuration files and basic structure in the attachment. App
servers connects cache server as client programmatically so no
configuration for that.


On Tue, Apr 4, 2017 at 6:15 PM, Andrey Mashenkov <andrey.mashenkov@gmail.com
> wrote:

> Hi Alper,
>
> Still not enough clear. Would you please share a grid configuration?
>
> On Tue, Apr 4, 2017 at 4:45 PM, Alper Tekinalp <alper@evam.com> wrote:
>
>> Hi Andrey.
>>
>> The structure is a bit complicated. Let me try to explane: We have 2
>> types of applications: 1 is cache server which contains external caches, 1
>> is our application servers that runs the logic. Application servers also
>> creates and maintains caches. We have 2 application servers that connects
>> each other as servers. And these 2 application servers connects cache
>> server as clients. Application servers and cache server are 2 different
>> topologies.
>>
>> Sometime worker threads are stucked on:
>> at sun.misc.Unsafe.park(Native Method)
>>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>   at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>   at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>   at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>   at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>> 0(GridFutureAdapter.java:161)
>>   at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>> (GridFutureAdapter.java:119)
>>
>> on put or get operations either caches on application servers or while
>> accessing caches on cache server.
>>
>> Our cached on application servers are affinity based so we only access
>> records on local server.
>>
>>
>>
>>
>> On Tue, Apr 4, 2017 at 4:09 PM, Andrey Mashenkov <
>> andrey.mashenkov@gmail.com> wrote:
>>
>>> Hi Alper,
>>>
>>> Thread is waiting for result of remote get operation.
>>>
>>> Are you sure your server 1 is standalone and has its own topology? I
>>> can't understand, how 1 and 3 connects to 1 as client.
>>> Would you please clarify?
>>>
>>> On Tue, Apr 4, 2017 at 3:43 PM, Alper Tekinalp <alper@evam.com> wrote:
>>>
>>>> Hi.
>>>>
>>>> Can someone point me a direction that why a thread can stuck on the
>>>> trace above? Where should I look for? How can I investicate the issue?
>>>> Where should I look?
>>>>
>>>> On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <alper@evam.com>
>>>> wrote:
>>>>
>>>>> Hi all.
>>>>>
>>>>>
>>>>> We have 3 ignite servers. Server 1 works as standalone. Server 2 and
3
>>>>> connects eachother as server but connects server 1 as client. In a point
of
>>>>> the time server 3 got stucked at:
>>>>>
>>>>>   at sun.misc.Unsafe.park(Native Method)
>>>>>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>>>   at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>>>>   at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>>>>   at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>>>>   at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>>> 0(GridFutureAdapter.java:161)
>>>>>   at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>>> (GridFutureAdapter.java:119)
>>>>>   at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>>> atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
>>>>>   at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>>> .get(GridCacheAdapter.java:4665)
>>>>>   at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>>> .get(GridCacheAdapter.java:1388)
>>>>>   at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>>> .get(IgniteCacheProxy.java:1121)
>>>>>   at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
>>>>>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>>   at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>   at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>>> keInternal(CachePassthroughInvocationHandler.java:99)
>>>>>   at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>>> ke(CachePassthroughInvocationHandler.java:78)
>>>>>   at com.sun.proxy.$Proxy56.get(Unknown Source)
>>>>>
>>>>> when getting record from server 1. Long after that server 2 also got
>>>>> stucked at the same trace and also server 2 and server 3 disconnects
from
>>>>> each other.
>>>>>
>>>>> We investigated gc logs and there is not an unusual behaviour. One
>>>>> things is server 1 logs errors as follows when server 2 and server 3
>>>>> disconnects:
>>>>>
>>>>> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg
>>>>> - Received message without registered handler (will ignore)
>>>>> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
>>>>> [arr= true, ctx=false, start=0], partId=199, flags=1,
>>>>> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>>> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0,
>>>>> createTtl=0, accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a,
>>>>> locTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=2937],
>>>>> msgTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>>> cacheDesc=null]
>>>>> Registered listeners:
>>>>>
>>>>>
>>>>> Where should we look for the main cause of the problem? What can cause
>>>>> such a behaviour? There seems nothing wrong on server 1 logs etc.
>>>>>
>>>>> We use ignite 1.8.3.
>>>>>
>>>>> --
>>>>> Alper Tekinalp
>>>>>
>>>>> Software Developer
>>>>> Evam Streaming Analytics
>>>>>
>>>>> Atatürk Mah. Turgut Özal Bulv.
>>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>>> 34758 İSTANBUL
>>>>>
>>>>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
>>>>> www.evam.com.tr
>>>>> <http://www.evam.com>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alper Tekinalp
>>>>
>>>> Software Developer
>>>> Evam Streaming Analytics
>>>>
>>>> Atatürk Mah. Turgut Özal Bulv.
>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>> 34758 İSTANBUL
>>>>
>>>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
>>>> www.evam.com.tr
>>>> <http://www.evam.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>



-- 
Alper Tekinalp

Software Developer
Evam Streaming Analytics

Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL

Tel:  +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>

Mime
View raw message