ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu <maxn...@hotmail.com>
Subject Re: Can't increase the speed of loadCache() when increasing more Ignite node
Date Tue, 18 Oct 2016 12:55:51 GMT
Of course, it's not trivial... and changes on database are required (new
field on primary table (better) or new "extended partition table" 1to1
relationship with primary table (id primary table, partitionId)) but using
CacheStoreAdapter implementation it's not such as complex. I would do: 

1. overwrite "write" method on your CacheStoreAdapter implementation to
ensure new entries take proper partition when are written on database: with
ignite.affinity(cacheName).partition(for entryKey), implement insert/update
of new partition field on primary table or on "extended partition table"
(your will need to do 2 inserts on write, one on primary table with entity
and a second one on extended with partitionId)
2. overwrite loadCache(IgniteBiInClosure<Object, BinaryObject> clo,
Object... args) on your CacheStoreAdapter implementation with a flag arg to
allow load cache on full scan mode or on partition mode (using partitionId
field created on primary table or join with "extended partition table")
3. Call cache.load(fullScanFlag) for full scan mode.
4. Once loaded, cache.forEach... put same entry on cache to force re-write
with correct partition.
5. Now table (primary or "partition tabled") is updated with correct
6. From now you can call cache.load(null) (for convenience, by default
without params on partition mode) and each node will load its own
partitioned data

Take a look to Affinity-Collocation
(https://apacheignite.readme.io/docs/affinity-collocation) to improve
performance and other important recommendations when use sql joins with
partitioned caches

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-t-increase-the-speed-of-loadCache-when-increasing-more-Ignite-node-tp8336p8347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message