ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Ripken <r...@rmanet.com>
Subject Re: Is there a way to allow nodes to join an already started task?
Date Mon, 22 May 2017 19:08:50 GMT
Thanks for the feedback.
I've read the feature docs and the bug report.  In summary: adding nodes 
doesn't work.

On 5/22/2017 4:43 AM, Andrey Mashenkov wrote:
> Hi Ryan,
> Ignite has JobStealingCollisionSpi and JobStealingFailoverSpi for it. 
> See CollisionSpi [1] and FailoverSpi [2] documentation.
> However, there is an unresolved issue [3] that doesn't allow tasks to 
> be stolen by newly joined node.
> [1] https://apacheignite.readme.io/docs/job-scheduling
> [2] https://apacheignite.readme.io/docs/fault-tolerance
> [3] https://issues.apache.org/jira/browse/IGNITE-1267
> On Tue, May 16, 2017 at 11:04 PM, Ryan Ripken <ryan@rmanet.com 
> <mailto:ryan@rmanet.com>> wrote:
>     In the GridGain days I was able to add nodes to an already started
>     task.   Is there a way to do that in Ignite?  I occasionally have
>     nodes disconnect or crash in a custom native library. I'd like to
>     be able to restart those failed nodes or add additional nodes if
>     the compute isn't progressing as quickly as initially hoped.
>     If its not possible to add nodes to an already started task, are
>     there patterns or tricks that can be used to accomplish something
>     similar?
>     It seems like one trick might be to take the original task (100
>     jobs) and turn it into many more tasks (10) with fewer jobs (10)
>     per task.  The new list of tasks aren't started all at once but
>     staggered over time so that if additional nodes join (after task 1
>     has already started) they can contribute to the later tasks.
>     Its a crude solution but it seems like it would have to work.
>     Before I refactor my tasks and jobs to try it out I'm wondering if
>     someone has a better suggestion.  Is there a better way to
>     accomplish something similar?
>     Thanks!
> -- 
> Best regards,
> Andrey V. Mashenkov

View raw message