ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@gridgain.com>
Subject Re: Problem with CacheLoadOnlyStoreAdapter
Date Tue, 17 Nov 2015 09:08:25 GMT
Hi Jakub,

I tried to reproduce your issue writing an advanced test but failed to 
catch any issue related to CacheLoadOnlyStoreAdapter.
In addition, looking at the code of this adapter I don't see any reason 
that can lead to the situation when store.parse() method is not called 
for every object retrieved from store.inputIterator().

What CacheLoadOnluStoreAdapter parameters do you use that let to catch 
the situation when store.parse() is not called for every object?

Please try to enable assertions by passing "-ea" parameter to VM 
parameters list. Probably some assertion has raised.

BTW, some comments in regards to your code:

1) You don't need to mark "inputIterator" and "parse" with 
"synchronized". These methods are already thread-safe.

2) You don't need to copy content of the collection returned from 
"loadDataFromDataBase" to CopyOnWriteArrayList. Just return 
collection.iterator() from "inputIterator". The store already works with 
the iterator in a thread-safe manner.

Regards,
Denis

On 11/16/2015 6:03 PM, jakub.martyna.comarch wrote:
> Sorry for reposting but I was not subscribed to mailing list
>
>
> jakub.martyna.comarch wrote
>> Hi,
>> We are trying to run ignite cache on 2 nodes with
>> CacheLoadOnlyStoreAdapter, but we have encountered problem. After loading
>> data ( cache.loadCache(null, (Object[]) null) ) we are checking if all
>> data are in cache. Unfortunately some objects are missing. The set of
>> missing objects is every time different.
>>
>> We store many different data types in single cache, all implement common
>> interface. As key we are using:
>>
>> public class SpaceAffinityKey implements Serializable {
>> 	private Long id;
>> 	private String className;
>> 	@AffinityKeyMapped
>> 	private Long routingId;
>>
>>          //equals & hash genereted from id and className
>> }
>>
>> class MyCacheLoader extends CacheLoadOnlyStoreAdapter {
>> 	protected synchronized Iterator&lt;Object[]&gt; inputIterator(Object...
>> args) {
>> 		List&lt;Object[]&gt; allObjects = new CopyOnWriteArrayList<>();
>>
>> 		for (String clazz: allTypesToLoad) {
>> 			Collection data = loadDataFromDataBase(clazz);
>> 			allObjects.addAll();
>> 			//log loaded data
>> 		}
>>
>> 		return allObjects.iterator();
>> 	}
>> 	
>> 	@Override
>> 	protected synchronized IgniteBiTuple&lt;SpaceAffinityKey,
>> IdentifiableById&gt; parse(Object[] rec, Object... args) {
>> 		IdentifiableById obj = (IdentifiableById) rec[0];
>> 		long routing = (long) rec[1];
>> 		//log parsed data
>> 		return new IgniteBiTuple&lt;SpaceAffinityKey, IdentifiableById&gt;(
>> 				new SpaceAffinityKey(obj.getId(), obj.getClass().getName(), routing),
>> obj);
>> 	}
>> }
>>
>> All data from DB are loaded in inputIterator.  In parse some objects are
>> missing.
>>
>> Changing configuration of CacheLoadOnlyStoreAdapter to threads=1
>> batchSize=1 queueSize=500000 fixes the problem. But it's slow. Moreover it
>> suggests that problem was during precessing batched data from loading
>> queues, but ignite had not logged any error.
>>
>> What can I do to improve load performance?
>>
>> Regards,
>> Jakub Martyna
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Problem-with-CacheLoadOnlyStoreAdapter-tp1966p1968.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Mime
View raw message