From dev-return-46946-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Aug 9 16:16:37 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id C55B918063F for ; Fri, 9 Aug 2019 18:16:36 +0200 (CEST) Received: (qmail 83308 invoked by uid 500); 9 Aug 2019 16:16:35 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 83295 invoked by uid 99); 9 Aug 2019 16:16:35 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Aug 2019 16:16:35 +0000 Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 26EDB845B for ; Fri, 9 Aug 2019 16:16:35 +0000 (UTC) Received: by mail-lj1-f178.google.com with SMTP id k18so92561317ljc.11 for ; Fri, 09 Aug 2019 09:16:35 -0700 (PDT) X-Gm-Message-State: APjAAAUrtvAxwMTgOAzwIN5EZ2jJQQ6WPDaZ6hniMnkfqA9rG+di4ZPW xb5ARpmSsVrr96QY1Da4SPspBXIR+3VBDHtxMw== X-Google-Smtp-Source: APXvYqxzDkL0LYYqwi4blkmRRpg0RyzAo7oq9OUoWnaGukyINh9NEEUk9gTV9tXi2Hy6nvbX70otaWKvhKh38JVK0XY= X-Received: by 2002:a2e:80d6:: with SMTP id r22mr11773074ljg.83.1565367394062; Fri, 09 Aug 2019 09:16:34 -0700 (PDT) MIME-Version: 1.0 References: <09da6b7bb9924d0ea04ad4e5ba0f4461@uk.bnpparibas.com> <1555325172243-0.post@n6.nabble.com> <658c837ea0544c2ba92c826b9af0e3ce@uk.bnpparibas.com> <1556186353304-0.post@n6.nabble.com> <1556783567878-0.post@n6.nabble.com> In-Reply-To: From: Pavel Tupitsyn Date: Fri, 9 Aug 2019 19:16:22 +0300 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang To: dev Content-Type: multipart/alternative; boundary="000000000000775617058fb17f75" --000000000000775617058fb17f75 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Well, you can't just take any running thread and make it run your code instead, right? On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev wrote: > Hello! > > Why can't we dispatch those callbacks in the caller thread? > > Regards, > -- > Ilya Kasnacheev > > > =D0=BF=D1=82, 9 =D0=B0=D0=B2=D0=B3. 2019 =D0=B3. =D0=B2 09:50, Pavel Tupi= tsyn : > > > =D0=98=D0=B2=D0=B0=D0=BD, > > > > The fix is to dispatch those callbacks (future listeners) to a differen= t > > thread pool, not sure which one though. > > If I would do a .NET-only fix, I would use the default thread pool (non > > Ignite-specific), for Java-side there is no such thing as I understand. > > > > Yes, let's have a ticket to track the issue. > > > > On Fri, Aug 9, 2019 at 9:17 AM =D0=9F=D0=B0=D0=B2=D0=BB=D1=83=D1=85=D0= =B8=D0=BD =D0=98=D0=B2=D0=B0=D0=BD > wrote: > > > > > Ilya, Pavel, > > > > > > Do we a have a proposal how to fix the root cause of the problem? > > > Should we a have a ticket for it? > > > > > > =D1=81=D1=80, 7 =D0=B0=D0=B2=D0=B3. 2019 =D0=B3. =D0=B2 17:48, Ilya K= asnacheev >: > > > > > > > > Hello! > > > > > > > > I think we should definitely stop running futures out of striped > pool, > > > > while holding any cache logs (stripe thread counts as one). > > > > > > > > Regards, > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > =D1=81=D1=80, 7 =D0=B0=D0=B2=D0=B3. 2019 =D0=B3. =D0=B2 17:20, Pave= l Tupitsyn : > > > > > > > > > Yes, this can be done purely on .NET side, which is an option tha= t > I > > > > > consider. > > > > > However, the root problem is on Java side, and I believe that we > > > should fix > > > > > the root problem. > > > > > > > > > > > violate some of Ignite assumptions: that we never run user code > > from > > > > > certain thread pools > > > > > We actually do run user code from Ignite thread pools: > > > > > > > > > > cache.getAsync(1).listen(fut -> > > > > > System.out.println("Get operation completed [value=3D" + > fut.get() > > + > > > > > ']')); > > > > > > > > > > `println` here is executed on the striped pool. This is stated in > the > > > > > docs that I linked above. > > > > > > > > > > Users have to be aware of this and they have to be very careful > with > > > > > every future listener. IMO, this is a tricky gotcha and a bad > > > > > usability. > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev < > > > ilya.kasnacheev@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Hello! > > > > > > > > > > > > + dev@ > > > > > > > > > > > > I think the current behavior, where .Net callbacks may be run > from > > > > > striped > > > > > > pool, violate some of Ignite assumptions: that we never run use= r > > code > > > > > from > > > > > > certain thread pools (like sys-stripe) and that we try to limit > > > options > > > > > of > > > > > > running user-supplied code from our internals. > > > > > > > > > > > > I think that future versions of .Net integration should remove > the > > > > > ability > > > > > > of async callbacks to be called from non-user threads, even if = it > > can > > > > > lead > > > > > > to performance degradation in some cases. I suggest removing th= is > > > mode, > > > > > if > > > > > > possible, while keeping only the safe one, where internal threa= ds > > > are not > > > > > > waiting upon completion of user code. > > > > > > > > > > > > In this case my issue IGNITE-12033 could be used to track this > > work. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > Regards, > > > > > > -- > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > =D1=81=D1=80, 7 =D0=B0=D0=B2=D0=B3. 2019 =D0=B3. =D0=B2 01:47, = Pavel Tupitsyn >: > > > > > > > > > > > >> Sorry guys, I've completely missed this thread, and the topic = is > > > very > > > > > >> important. > > > > > >> > > > > > >> First, a simple fix for the given example. Add the following o= n > > the > > > > > first > > > > > >> line of Main: > > > > > >> SynchronizationContext.SetSynchronizationContext(new > > > > > >> ThreadPoolSynchronizationContext()); > > > > > >> > > > > > >> And put the ThreadPoolSynchronizationContext class somewhere: > > > > > >> class ThreadPoolSynchronizationContext : SynchronizationContex= t > > > > > >> { > > > > > >> // No-op. > > > > > >> } > > > > > >> > > > > > >> > > > > > >> Now, detailed explanation. The problem exists forever in Ignit= e > > and > > > is > > > > > >> mentioned in the docs briefly [1]. > > > > > >> Also mentioned in .NET docs (I've updated them a bit) [2]. > > > > > >> > > > > > >> Breakdown: > > > > > >> * Ignite (Java side) runs async callbacks (continuations) on > > system > > > > > >> threads, and those threads have limitations (you should not ca= ll > > > Ignite > > > > > >> APIs from them in general) > > > > > >> * Ignite.NET wraps async operations into native .NET Tasks > > > > > >> * Usually `await ...` call in .NET will continue execution on > the > > > > > >> original Thread (simply put, actually it is more complex), so > > Ignite > > > > > system > > > > > >> thread issue is avoided > > > > > >> * However, Console applications have no > `SynchronizationContext`, > > > so the > > > > > >> continuation can't be dispatched to original thread, and is > > > executed on > > > > > >> current (Ignite) thread > > > > > >> * Setting custom SynchronizationContext fixes the issue: all > async > > > > > >> continuations will be dispatched to .NET thread pool and never > > > executed > > > > > on > > > > > >> Ignite threads > > > > > >> > > > > > >> However, dispatching callbacks to a different thread causes > > > performance > > > > > >> hit, and Ignite favors performance over usability right now. > > > > > >> So it is up to the user to configure desired behavior. > > > > > >> > > > > > >> Let me know if you need more details. > > > > > >> > > > > > >> Thanks > > > > > >> > > > > > >> [1] https://apacheignite.readme.io/docs/async-support > > > > > >> [2] > https://apacheignite-net.readme.io/docs/asynchronous-support > > > > > >> > > > > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev < > > > > > ilya.kasnacheev@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >>> Hello! > > > > > >>> > > > > > >>> I have filed a ticket about this issue so it won't get lost. > > > > > >>> https://issues.apache.org/jira/browse/IGNITE-12033 > > > > > >>> > > > > > >>> Regards, > > > > > >>> -- > > > > > >>> Ilya Kasnacheev > > > > > >>> > > > > > >>> > > > > > >>> =D1=87=D1=82, 2 =D0=BC=D0=B0=D1=8F 2019 =D0=B3. =D0=B2 10:53,= Barney Pippin < > > > > > james.prince@uk.bnpparibas.com > > > > > >>> >: > > > > > >>> > > > > > >>>> Thanks for the response Ilya. Did you get a chance to look a= t > > this > > > > > >>>> Pavel? > > > > > >>>> Thanks. > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> -- > > > > > >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > > > > > >>>> > > > > > >>> > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > --000000000000775617058fb17f75--