ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@gridgain.com>
Subject Re: Data Loading Performance Issue
Date Wed, 18 Nov 2015 08:59:29 GMT
Hi Deha,

Thanks for the additional info. However I didn't manage to open your config
file.

In any case I have the following thoughts basing on the current info.

/Currently most important issue is, overall batch is running faster in my
local PC (Windows 7, 8CPU/32GB RAM machine) than more powerful Linux server
(64 CPU and 1TB RAM, which also has a faster network to Oracle DB)
/

Since you're populating a LOCAL cache and if to take into account that your
local machine and the servers use the same configuration and virtual machine
options the only reason that should lead to such big performance differences
is the speed with which the data retrieved from the DB.

Yeah, you measured connection speed using 'ping' utility but it has nothing
to do with actual data retrieving from the DB.

I suggest you doing the following steps:
- measure the time that it takes to get the data from DB. It should be
different on your local and server machines;
- check settings of your Oracle/JDBC connector on the server side. 


/Yes my cache structure IgniteCache<Integer, ArrayList&lt;MyBusinessObj>>,
as I needed to process records that has certain common attributes together,
these are like trade positions that has certain common attributes like date,
acct, currency etc. After reading into a cache during the proecessing stage
each group split into more granular records like 10 records become 1000
records and then I aggregate (group by) them so number of records shrinks to
500. then I directly write these records into some feed file. During the
interim processing/computation the don't get stored back to Ignite cache
just stays in regular Java memory till got flush into file. /

Sounds complex for me. Fully share Val's concerns that in the future you
will face with the performance issues when these data needs to be processed,
requested, updated.

My suggestion is to store different business objects in different caches.
After that you can easily use SQL queries with joins to get the data you
need from the cache.
https://apacheignite.readme.io/docs/sql-queries

/You can utilize CacheLoadOnlyStoreAdapter for this. 
diopek:
Is there any working example you can point me to ? /

Yes, this is a great suggestion from Val. 

We don't have a dedicated example for this kind of store. However, you may
refer to some tests that shows how to use it -
GridCacheLoadOnlyStoreAdapterSelfTest.

Start using this store with default settings and adjust them depending on
your the characteristics of a machine.

Regards,
Denis



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Loading-Performance-Issue-tp1958p2003.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message