ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vkulichenko <valentin.kuliche...@gmail.com>
Subject Re: Data Loading Performance Issue
Date Tue, 17 Nov 2015 20:33:38 GMT
Hi Deha,

diopek wrote
> 1)Highest priority one, ironically, my local machine batch run time is
> much lower 42 mins (local) vs. 2hr 56 mins (linux) both running on single
> node eve though it is less powerful and I have other other apps running at
> the same time such as Eclipse IDE, SQLdeveloper, Outlook, Word, UNIX box
> is dedicated app server no other applications running on that server.

Can you please clarify what you mean by "batch run time"? Is it somehow
connected to data loading via the store or it's a different issue?

diopek wrote
> 2) Second one is in general loading data from DB to Java Collection is
> much faster than loading into Ignite cache by factor of ~6. Which is not a
> show stopper but if we need to resolve the first problem as highest
> priority. I appreciate your help on this. Thanks

I noticed that you put some lists instead of individual entries into the
cache. What is the size of these lists? My suspicion is that the most time
is spent for the serialization of the values (JCache spec has pass-by-value
semantics, so we have to do this even in LOCAL cache).

I would suggest to store a value per DB row to avoid duplicate data and
therefore duplicate serializations. It also looks like loading the data in
multithreaded fashion may be helpful - execute the query first and then do
DB row parsing and saving into cache in several parallel threads. You can
utilize CacheLoadOnlyStoreAdapter for this.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Loading-Performance-Issue-tp1958p1993.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message