ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jakub.martyna.comarch" <jakub.mart...@comarch.pl>
Subject Re: Problem with CacheLoadOnlyStoreAdapter
Date Mon, 16 Nov 2015 15:03:48 GMT
Sorry for reposting but I was not subscribed to mailing list


jakub.martyna.comarch wrote
> Hi,
> We are trying to run ignite cache on 2 nodes with
> CacheLoadOnlyStoreAdapter, but we have encountered problem. After loading
> data ( cache.loadCache(null, (Object[]) null) ) we are checking if all
> data are in cache. Unfortunately some objects are missing. The set of
> missing objects is every time different.
> 
> We store many different data types in single cache, all implement common
> interface. As key we are using:
> 
> public class SpaceAffinityKey implements Serializable {
> 	private Long id;
> 	private String className;
> 	@AffinityKeyMapped
> 	private Long routingId;
> 
>         //equals & hash genereted from id and className
> }
> 
> class MyCacheLoader extends CacheLoadOnlyStoreAdapter {
> 	protected synchronized Iterator&lt;Object[]&gt; inputIterator(Object...
> args) {
> 		List&lt;Object[]&gt; allObjects = new CopyOnWriteArrayList<>();
> 
> 		for (String clazz: allTypesToLoad) {
> 			Collection data = loadDataFromDataBase(clazz);        
> 			allObjects.addAll();
> 			//log loaded data
> 		}
> 
> 		return allObjects.iterator();
> 	}
> 	
> 	@Override
> 	protected synchronized IgniteBiTuple&lt;SpaceAffinityKey,
> IdentifiableById&gt; parse(Object[] rec, Object... args) {
> 		IdentifiableById obj = (IdentifiableById) rec[0];
> 		long routing = (long) rec[1];
> 		//log parsed data
> 		return new IgniteBiTuple&lt;SpaceAffinityKey, IdentifiableById&gt;(
> 				new SpaceAffinityKey(obj.getId(), obj.getClass().getName(), routing),
> obj);
> 	}
> }
> 
> All data from DB are loaded in inputIterator.  In parse some objects are
> missing.
> 
> Changing configuration of CacheLoadOnlyStoreAdapter to threads=1
> batchSize=1 queueSize=500000 fixes the problem. But it's slow. Moreover it
> suggests that problem was during precessing batched data from loading
> queues, but ignite had not logged any error.
> 
> What can I do to improve load performance?
> 
> Regards,
> Jakub Martyna





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Problem-with-CacheLoadOnlyStoreAdapter-tp1966p1968.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message