ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vkulichenko <valentin.kuliche...@gmail.com>
Subject Re: Super slow data loading performance when more nodes added
Date Tue, 25 Aug 2015 03:39:13 GMT
Sorry for the confusion. I actually meant that in your scenario update
operations are not collocated and will imply network trips (unlike Hadoop
which will write data locally unless you run out of space). So the way you
split the process most likely will not give you performance improvement -
you're minimizing the amount of data transferred from S3 to nodes, but the
greater part of it will still be transferred between nodes.

Since you're loading from the remote storage, I think the best way is to use
CacheStore to load the data, like described in [1]. In this case each node
will have to read the whole file, but all updates will be local.

Let me know if it helps.

[1] https://apacheignite.readme.io/docs/data-loading#ignitecacheloadcache



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Super-slow-data-loading-performance-when-more-nodes-added-tp1105p1119.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message