ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Kornev <andrewkor...@hotmail.com>
Subject RE: Computation on NodeEntries
Date Fri, 11 Dec 2015 18:05:11 GMT
Great suggestion! Thank you, Yakov!

Just one more question. :) Let's say the scan job is running node A and processing partition
42. At the same time, a new node B joins and partition 42 needs to be moved to this node.
What will happen to my scan query that is still running on node A and iterating over the partition's
entries? Would it complete processing the entire partition despite the change of ownership?
Or, would the query terminate at some arbitrary point once the partition ownership transfer
has completed?

Thanks a lot!
Andrey

Date: Fri, 11 Dec 2015 16:06:16 +0300
Subject: Re: Computation on NodeEntries
From: yzhdanov@apache.org
To: user@ignite.apache.org

Guys, I would do the following:
1. Map all my partitions to nodes: org.apache.ignite.cache.affinity.Affinity#mapPartitionsToNodes2.
Send jobs (with its list of partitions) to each node using map returned on step13. Job may
be like:new Runnable() {
    @Override public void run() {
        for (Integer part : parts) {
            Iterator<Cache.Entry<Object, Object>> it = cache.query(new ScanQuery<>(part)).iterator();
            
            // do the stuff...
        }
        
    }
};
This may result in network calls for some worst cases when topology changes under your feet,
but even in this case this should work.--Yakov

2015-12-11 2:13 GMT+03:00 Andrey Kornev <andrewkornev@hotmail.com>:



Dmitriy,

Given the approach you suggested below, what would be your recommendation for dealing with
cluster topology changes while the iteration is in progress? An obvious one I can think of
is to 
- somehow detect the change, 
- cancel the tasks on all the nodes 
- wait until the rebalancing is finished and
- restart the computation.

Are there any other ways? Ideally, I'd like to have the "exactly-once" execution semantics.

Thanks
Andrey
 		 	   		  
Mime
View raw message