ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Mekhanikov <dmekhani...@gmail.com>
Subject Re: ignite.compute(ClusterGroup) is broken??
Date Tue, 05 Dec 2017 10:31:23 GMT
Chris,

Tasks are sent to all nodes, that return from map() method, regardless of
the provided ClusterGroup.

ClusterGroup only affects a list of nodes, that are specified in *subgrid*
parameter. So, it's your responsibility to assign tasks to nodes, that
match your needs. If you want to send them to nodes from a cluster group
only, then you should consider only nodes from *subgrid* list.

Denis

пн, 4 дек. 2017 г. в 22:42, Chris Berry <chriswberry@gmail.com>:

> Hi Denis,
>
> Thank you for responding.
>
> >>> Can you provide a short summary of what you are trying to achieve?
>
> The problem is this.
>
> Ignite begins sending compute traffic to a Node (i.e. ComputeJobs), before
> that Node is truly ready to take traffic. Basically, as soon as Ignite
> starts and is balanced on a Node, it begins receiving compute traffic.
>
> This is wrong.
> It means that as a Node starts up, it is slammed with traffic it is not
> ready to handle yet.
> So. We must reject that traffic until we are ready (throw an Exception).
> Which, then, fails that ComputeJob over to another Node.
> This, of course, works. But it is costly, in terms of system throughput as
> Nodes come and go in AWS.
>
> We need the ability to notify a Node that it is ready to handle compute
> load.
> Instead of Ignite *assuming* that the Node is ready as soon as Ignite is
> started.
>
> It is a classic chicken-and-egg problem.
> We need Ignite started and rebalanced, because we need to interact w/ some
> Ignite caches.
> Thus, we need to start Ignite before we can finish enabling the app.
> In addition, many other things at startup things happen in the background,
> such as Kafka streaming in data to some caches, etc.
> And when all of this is complete, we toggle an “Alive switch” that
> indicates
> that the Node is ready for business.
>
> >>> The main thing that I don't understand is why you need a map method?
> >>> This method is supposed to be used as a part of a map-reduce pipeline.
> >>> It is aimed to split one big task into many small jobs.
>
> This is exactly how we are using it.
>
> But the question at hand here is about whether;
> ignite.cluster.compute(ClusterGroup) is broken or not.
>
> Because my logs clearly show me that even though I am absolutely certain
> that a ClusterGroup, passed to the compute() method, does NOT contain a
> particular Node.
> I can see that Node receiving ComputeJobs.
>
> I do not see how that can be possible.
>
> Thanks,
> -- Chris
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Mime
View raw message