ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Tupitsyn <ptupit...@apache.org>
Subject Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Date Fri, 09 Aug 2019 06:50:00 GMT
Иван,

The fix is to dispatch those callbacks (future listeners) to a different
thread pool, not sure which one though.
If I would do a .NET-only fix, I would use the default thread pool (non
Ignite-specific), for Java-side there is no such thing as I understand.

Yes, let's have a ticket to track the issue.

On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван <vololo100@gmail.com> wrote:

> Ilya, Pavel,
>
> Do we a have a proposal how to fix the root cause of the problem?
> Should we a have a ticket for it?
>
> ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <ilya.kasnacheev@gmail.com>:
> >
> > Hello!
> >
> > I think we should definitely stop running futures out of striped pool,
> > while holding any cache logs (stripe thread counts as one).
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <ptupitsyn@apache.org>:
> >
> > > Yes, this can be done purely on .NET side, which is an option that I
> > > consider.
> > > However, the root problem is on Java side, and I believe that we
> should fix
> > > the root problem.
> > >
> > > > violate some of Ignite assumptions: that we never run user code from
> > > certain thread pools
> > > We actually do run user code from Ignite thread pools:
> > >
> > > cache.getAsync(1).listen(fut ->
> > >     System.out.println("Get operation completed [value=" + fut.get() +
> > > ']'));
> > >
> > > `println` here is executed on the striped pool. This is stated in the
> > > docs that I linked above.
> > >
> > > Users have to be aware of this and they have to be very careful with
> > > every future listener. IMO, this is a tricky gotcha and a bad
> > > usability.
> > >
> > >
> > > Thoughts?
> > >
> > >
> > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > + dev@
> > > >
> > > > I think the current behavior, where .Net callbacks may be run from
> > > striped
> > > > pool, violate some of Ignite assumptions: that we never run user code
> > > from
> > > > certain thread pools (like sys-stripe) and that we try to limit
> options
> > > of
> > > > running user-supplied code from our internals.
> > > >
> > > > I think that future versions of .Net integration should remove the
> > > ability
> > > > of async callbacks to be called from non-user threads, even if it can
> > > lead
> > > > to performance degradation in some cases. I suggest removing this
> mode,
> > > if
> > > > possible, while keeping only the safe one, where internal threads
> are not
> > > > waiting upon completion of user code.
> > > >
> > > > In this case my issue IGNITE-12033 could be used to track this work.
> > > >
> > > > WDYT?
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <ptupitsyn@apache.org>:
> > > >
> > > >> Sorry guys, I've completely missed this thread, and the topic is
> very
> > > >> important.
> > > >>
> > > >> First, a simple fix for the given example. Add the following on the
> > > first
> > > >> line of Main:
> > > >> SynchronizationContext.SetSynchronizationContext(new
> > > >> ThreadPoolSynchronizationContext());
> > > >>
> > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > > >> {
> > > >>     // No-op.
> > > >> }
> > > >>
> > > >>
> > > >> Now, detailed explanation. The problem exists forever in Ignite and
> is
> > > >> mentioned in the docs briefly [1].
> > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > >>
> > > >> Breakdown:
> > > >> * Ignite (Java side) runs async callbacks (continuations) on system
> > > >> threads, and those threads have limitations (you should not call
> Ignite
> > > >> APIs from them in general)
> > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > >> * Usually `await ...` call in .NET will continue execution on the
> > > >> original Thread (simply put, actually it is more complex), so Ignite
> > > system
> > > >> thread issue is avoided
> > > >> * However, Console applications have no `SynchronizationContext`,
> so the
> > > >> continuation can't be dispatched to original thread, and is
> executed on
> > > >> current (Ignite) thread
> > > >> * Setting custom SynchronizationContext fixes the issue: all async
> > > >> continuations will be dispatched to .NET thread pool and never
> executed
> > > on
> > > >> Ignite threads
> > > >>
> > > >> However, dispatching callbacks to a different thread causes
> performance
> > > >> hit, and Ignite favors performance over usability right now.
> > > >> So it is up to the user to configure desired behavior.
> > > >>
> > > >> Let me know if you need more details.
> > > >>
> > > >> Thanks
> > > >>
> > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> > > >>
> > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > > ilya.kasnacheev@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hello!
> > > >>>
> > > >>> I have filed a ticket about this issue so it won't get lost.
> > > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > > >>>
> > > >>> Regards,
> > > >>> --
> > > >>> Ilya Kasnacheev
> > > >>>
> > > >>>
> > > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > > james.prince@uk.bnpparibas.com
> > > >>> >:
> > > >>>
> > > >>>> Thanks for the response Ilya. Did you get a chance to look
at this
> > > >>>> Pavel?
> > > >>>> Thanks.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > > >>>>
> > > >>>
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message