ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sri Ganesh V <vsrigan...@yahoo.co.in>
Subject Isuue with Multi-threaded approach to Ignite data streamer
Date Wed, 19 Oct 2016 17:03:47 GMT
Hi,

We use Partitioned cache across 3 server nodes.

Cache Memory Mode : off-heap

We are trying to load cache using ignite data streamer.
We have an ignite client pool and use different clients to create streamer
for loading the cache.

We fetch data from DB in multiple threads and push the same to the streamer.

Please find the code below -


	
	public void run() {

		String beanClassName = CacheBeanFactory.getBean(this.cacheName);

		IgniteDataStreamer stmr = null;
		IgniteClientConnectionPool igniteClientPool = null;
		Ignite ignite = null;
		 try {
			ICacheLoadClient clientDao = new CacheLoadClientImpl();
			
			// Ignite client pool
			igniteClientPool = IgniteClientConnectionPool.getInstance();
			ignite = igniteClientPool.getClient();
			
			stmr = ignite.dataStreamer(this.cacheName);
			
			

			Map<String, Object> addToCacheMap = clientDao.loadCache(// Fetch records
from DB//);

			
		
			if (addToCacheMap.size() != 0)
			{
				
				
				Iterator<String> iter = addToCacheMap.keySet().iterator();
				String key ="";
				while(iter.hasNext())
				{
					key = iter.next();
					stmr.addData(key,addToCacheMap.get(key));
				}
				
				
				
				
			}}
		
		catch (Exception e) {
			e.printStackTrace();
			
		} finally {
			if (stmr != null)
			{
				System.out.println("<Closing Streamer>");
				stmr.close();
			}
			if(igniteClientPool!=null && ignite!=null )
			igniteClientPool.free(ignite);
			
		}

      }

Currently there are 20 concurrent threads trying to load the same cache via
different streamers.

ISSUE : We face the issue of thread getting stuck trying to add data to the
streamer and the entire application becomes standstill.

At times we have seen scenarios were in after 15 -20 min the load process
continues. 

It would be great if we get to know the best way of using streamers in an
multi-threaded set-up to load a cache.

Thanks,
Ganesh







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Isuue-with-Multi-threaded-approach-to-Ignite-data-streamer-tp8362.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message