ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Kornev <andrewkor...@hotmail.com>
Subject RE: Computation on NodeEntries
Date Sun, 13 Dec 2015 06:08:51 GMT
Yakov,

If partions do not migrate while they are being iterated over, wouldn't it then suffice to
simply execute a single ScanQuery with its isLocal set to true? My reasoning here is that
the scan would create an iterator for all affinity partitions, thus preventing their migration.
If it's not the case, how would then a local ScanQuery behave in presence of topology changes?
Also, what's the best way to handle topology changes while using the SqlQuery rather than
ScanQuery? Basically, it's the same use case, only instead of scanning the entire partition
I'd like to first filter the cache entries using a query.
ThanksAndrey
    _____________________________
From: Yakov Zhdanov <yzhdanov@apache.org>
Sent: Friday, December 11, 2015 10:55 AM
Subject: RE: Computation on NodeEntries
To:  <user@ignite.apache.org>


    

Partition will not migrate if local or remote iterator is not finished/closed.      On Dec
11, 2015 21:05, "Andrey Kornev" <   andrewkornev@hotmail.com> wrote:   
                   Great suggestion! Thank you, Yakov!      
      
Just one more question. :) Let's say the scan job is running node A and processing partition
42. At the same time, a new node B joins and partition 42 needs to be moved to this node.
What will happen to my scan query that is still running on node A and iterating over the partition's
entries? Would it complete processing the entire partition despite the change of ownership?
Or, would the query terminate at some arbitrary point once the partition ownership transfer
has completed?      
      
Thanks a lot!      
Andrey      
      
             Date: Fri, 11 Dec 2015 16:06:16 +0300       
Subject: Re: Computation on NodeEntries       
From:        yzhdanov@apache.org       
To:        user@ignite.apache.org       
       
               Guys, I would do the following:                 
                         1. Map all my partitions to nodes: org.apache.ignite.cache.affinity.Affinity#mapPartitionsToNodes
                        2. Send jobs (with its list of partitions) to each node using map
returned on step1                         3. Job may be like:                         new
Runnable() {
    @Override public void run() {
        for (Integer part : parts) {
            Iterator<Cache.Entry<Object, Object>> it = cache.query(new ScanQuery<>(part)).iterator();
            
            // do the stuff...
        }
        
    }
};
         This may result in network calls for some worst cases when topology changes under
your feet, but even in this case this should work.                         
                                          --Yakov                                        
                   2015-12-11 2:13 GMT+03:00 Andrey Kornev           <andrewkornev@hotmail.com>:
         
                                               Dmitriy,             
             
Given the approach you suggested below, what would be your recommendation for dealing with
cluster topology changes while the iteration is in progress? An obvious one I can think of
is to              
- somehow detect the change,              
- cancel the tasks on all the nodes              
- wait until the rebalancing is finished and             
- restart the computation.             
             
Are there any other ways? Ideally, I'd like to have the "exactly-once" execution semantics.
            
             
Thanks             
Andrey             
                                                                                 


  
Mime
View raw message