ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Proper collocation of computations and data.
Date Thu, 20 Apr 2017 08:05:49 GMT
I think the following should be done:

   - AffinityKeyMapped annotation should be fixed if it's broken
   (obviously).
   - There should be a way to specify cache name in addition to key
   (another annotation?).
   - Semantics of a job or closure executed with this annotation should be
   the same as for affinityRun/Call, with all guarantees for reserved
   partitions, etc.

Alexey, this would cover all the cases you described, right?

-Val

On Wed, Apr 19, 2017 at 9:16 PM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:

> affinityRun will work only for single closure.
>
> But I need to have the same functionality for all jobs in my compute task,
> with support for task sessions and other benefits from map-reduce API.
>
>
>
> 2017-04-19 22:07 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
>
> > In this case, you should be using this API on IgniteCompute:
> >
> > affinityRun(Collection<String> cacheNames, int partId, IgniteRunnable
> job)
> >
> > This will ensure that the partition is not migrated while the computation
> > is in progress. Why is this method insufficient?
> >
> > As far as IgniteCompute deprecating the "withAsync" API, the async
> behavior
> > will now be supported through direct async API invocations now, so the
> > support for async behavior is still there.
> >
> > D.
> >
> > On Wed, Apr 19, 2017 at 11:59 AM, Alexei Scherbakov <
> > alexey.scherbakoff@gmail.com> wrote:
> >
> > > Dmitriy,
> > >
> > > I know I could, but it requires too much work and messing with various
> > APIs
> > > for correct routing/failover.
> > >
> > > What about automatic partition reservation for consitency? How could I
> > > achieve what using only affinity API ?
> > >
> > > I think this should be available out of the box, as for
> affinityRun/Call.
> > >
> > > Moreover, I could say the same for affinityRun/Call. One can use
> affinity
> > > API and determine node to send a closure. Why bother with additional
> > > methods ?
> > >
> > > And javadoc in AffinityKeyMapped definitely should be fixed, because it
> > > refers to unexistent things.
> > >
> > > 2017-04-19 21:38 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
> > >
> > > > Alexey,
> > > >
> > > > Have you taken a look at the Affinity API in Ignite? It seems that it
> > has
> > > > all the functionality you may need to map partitions to nodes. You
> can
> > > take
> > > > that info, and use it to route your computations.
> > > >
> > > > https://ignite.apache.org/releases/latest/javadoc/org/
> > > > apache/ignite/cache/affinity/Affinity.html
> > > >
> > > > D.
> > > >
> > > > On Wed, Apr 19, 2017 at 10:09 AM, Alexei Scherbakov <
> > > > alexey.scherbakoff@gmail.com> wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > Currently I'm looking into the problem how to properly deliver
> > > > computation
> > > > > to data in most efficient way.
> > > > >
> > > > > Basically I need to iterate over all cache partitions on all grid
> > > nodes,
> > > > > compute some function on each key-value pair and return aggregated
> > > result
> > > > > to a caller.
> > > > >
> > > > > This is a job for map-reduce API.
> > > > >
> > > > > But it seems where is no possibility to easily manage automatic
> > routing
> > > > and
> > > > > failover of compute jobs to data nodes containing specific
> > partitions.
> > > > >
> > > > > I found interesting paragraph in javadoc for @AffinityKeyMapped
> > > > annotation.
> > > > >
> > > > > Collocating Computations And Data
> > > > > It is also possible to route computations to the nodes where the
> data
> > > is
> > > > > cached. This concept is otherwise known as Collocation Of
> > Computations
> > > > And
> > > > > Data ....
> > > > >
> > > > > which makes strong sense for me.
> > > > >
> > > > > But in fact this is not working. Instead we only have automatic
> > > > routing(and
> > > > > failover) for special cases: affinityCall and affinityRun with
> > explicit
> > > > > partition. And it seems I can't longer use task sessions for them
> > with
> > > > > recent changes in Compute API (removed withAsync support)
> > > > >
> > > > > I think this is not OK and we should allow jobs to be automatically
> > > > routed
> > > > > to data if they have some annotation attached to them specifying
> > > > partition
> > > > > and cache names, same as for affinityCall/Run. Probably we should
> > > > introduce
> > > > > special task type for such workflows, something like
> > > AffinityComputeTask,
> > > > > without explicit mapping phase, for convenient usage.
> > > > >
> > > > > I'm willing to make a JIRA ticket for this.
> > > > >
> > > > > Thoughs ?
> > > > >
> > > > > --
> > > > >
> > > > > Best regards,
> > > > > Alexei Scherbakov
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message