ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Popov <tank2.a...@gmail.com>
Subject RE: Node failed to startup due to deadlock
Date Thu, 09 Nov 2017 14:26:01 GMT
Hi Naresh,

Unfortunately I don’t see any attachment, please send them again.

Can you clarify your node startup activity.
1) How do you lock the caches? IgniteCache.lock() or IgniteCache.lockAll() locks only a one
or batch of cache entries, not the whole Cache.
2) Do you really need some locks if the cache is just cleared
3) How do you perform a dataload? Via Streaming API or via CacheStore (i.e. IgniteCache.loadCache())?
4) I see that you have transactional Caches, how do you expect transaction to be finished
during new Node startup & cache cleanup?

Please note that Ignite Caches are distributed and self-balanced. You don’t need to reload
a Cache when a new node enters a cluster. The cluster will rebalance the caches it have. And
the new node will receive a part of Cache entries without any extra actions from your side.

Thank you,

From: naresh.goty
Sent: Thursday, November 9, 2017 9:54 AM
To: user@ignite.apache.org
Subject: RE: Node failed to startup due to deadlock

Hi Alexey,

Thank you for pointing out the problem with caches causing deadlocks.
I fixed the classpath pointing to 1.9, but the issue still exists.

Actually we could reproduce deadlock with a scenario when two nodes trying
to comeup during same time, and the first node holds lock on a cache. 
(attached sample code when run in the following order:
1) Start App.java and wait till the cache is locked
2) start App2.java, then we see deadlock on App2. 

We also tried with patch provided in (IGNITE-6380), but it is rejecting jobs
which are waiting.
ex=class o.a.i.compute.ComputeExecutionRejectedException: Pending topology
found - job execution within lock or transaction was canceled., hasRes=true,
isCancelled=false, isOccupied=true]
class org.apache.ignite.IgniteException: Remote job threw exception.

To summarize our node startup process:
1) Each node upon start, it will create or get the caches. 
2) Lock the caches
3) if caches not already loaded, then it will clear the caches, and perform
4) notify through ignite messaging about cache load activities
5) release lock on caches.

So, if multiple nodes are starting at the same time, all the nodes will try
to perform the above activities. During this process we will ensure that,
each cache will be loaded only once.  

Can you please confirm if the above usecase is supported with ignite?


Sent from: http://apache-ignite-users.70518.x6.nabble.com/

View raw message