ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolay Tikhonov <ntikho...@gridgain.com>
Subject Re: API for asynchronous execution.
Date Thu, 14 Apr 2016 10:58:35 GMT
We are close to completing IGNITE-2004 ticket.
As part this ticket was made the following changes on public API
- if callback has @IgniteAsyncCallback annotation then callback should be
run asynchronously
- these callbacks are executed in special pool (callback thread pool) which
is configured by IgniteConfiguration.asyncCallbackThreadPoolSize

Any comments on this?

On Wed, Mar 30, 2016 at 12:45 PM, Yakov Zhdanov <yzhdanov@gridgain.com>
wrote:

> I think this approach works unless user does not initiate number of
> concurrent cache operations greater than MSG_QUEUE_SIZE.  Where msg queue
> size default is 1024, but still configurable.
>
> Thanks!
> --
> Yakov Zhdanov, Director R&D
> *GridGain Systems*
> www.gridgain.com
>
> 2016-03-30 11:44 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:
>
> > Does it mean that if cache update rate is greater than filter execution
> > rate, then at some point we will stop reading messages from socket? If
> yes,
> > then it seems we still cannot execute cache operations:
> > 1) Filter starts cache operation for a key. Current node is backup for
> this
> > key.
> > 2) Cache message is sent to primary node
> > 3) Primary sends message back to current node.
> > 4) Message is never read because of backpressure. Cache operation and
> > filter never complete.
> >
> > Am I missing something?
> >
> > On Wed, Mar 30, 2016 at 11:23 AM, Yakov Zhdanov <yzhdanov@apache.org>
> > wrote:
> >
> > > Vladimir,
> > >
> > > Communication should stop reading from connection is there are too many
> > > unprocessed messages. Sender will be blocked on putting message to
> queue.
> > >
> > > --Yakov
> > >
> > > 2016-03-30 11:11 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:
> > >
> > > > Guys,
> > > >
> > > > Can you explain how backpressure control is implemented? What if
> event
> > > > arrival speed is greater than filter processing speed?
> > > >
> > > > On Wed, Mar 30, 2016 at 10:37 AM, Semyon Boikov <
> sboikov@gridgain.com>
> > > > wrote:
> > > >
> > > > > Andrey,
> > > > >
> > > > > I agree that current situation with threading in Ignite is very
> > > > > inconvenient when user callbacks execute some non-trivial code. But
> > > > > changing this to async dispatch is huge refactoring, even changing
> > this
> > > > > just for continuous queries callback is not so easy task.
> > > > >
> > > > > We can start with
> https://issues.apache.org/jira/browse/IGNITE-2004,
> > > and
> > > > > if
> > > > > more users complains arise we can think about changing others parts
> > of
> > > > > system.
> > > > >
> > > > > For now we need decisions for these points:
> > > > > - how to specify that callback should be run asynchronously
> (Nikolay
> > > > > suggested marker interface IgniteAsyncCallback, or
> > > @IgniteAsyncCallback)
> > > > > - where these callbacks are executed, AFAIK Nikolay added special
> > pool
> > > > > which is configured in IgniteConfiguration (something like
> > > > > IgniteConfiguration.asyncCallbackThreadPoolSize)
> > > > >
> > > > > Regards
> > > > >
> > > > >
> > > > > On Tue, Mar 29, 2016 at 10:45 PM, Andrey Kornev <
> > > > andrewkornev@hotmail.com>
> > > > > wrote:
> > > > >
> > > > > > Vladimir, Igniters
> > > > > >
> > > > > > Here are my 2 cents.
> > > > > >
> > > > > > The current situation with threading when it comes to executing
> > user
> > > > > > callbacks -- the CQ filters (either local or remote), the CQ
> > > listeners,
> > > > > the
> > > > > > event listeners, the messaging listeners, the entry processors
> > (did I
> > > > > miss
> > > > > > anything?) -- is pretty sad. The callbacks may get executed
on a
> > > system
> > > > > > pool's thread, public pool's, utility pool's, discovery worker
> > > thread,
> > > > > > application thread, to name a few. It causes a lot of grief
and
> > > > > suffering,
> > > > > > hard-to-fix races, dead locks and other bugs.
> > > > > >
> > > > > > I guess it's always possible to come up with a more or less
> > > reasonable
> > > > > > explanation to such predicament (which usually boils down to
"It
> is
> > > so
> > > > > > because this is how it's implemented"), but I, as a user, could
> not
> > > > care
> > > > > > less. I want consistency. I want all my callbacks (including
> Entry
> > > > > > Processors!) to be executed on the public pool's threads, to
be
> > > > precise.
> > > > > > This is not the first time I complain about this, and I really
> > think
> > > > it's
> > > > > > time to fix this mess.
> > > > > >
> > > > > > For a good example of how to implement ordered async dispatch
of
> > > > > callbacks
> > > > > > on large scale, one only needs to look at Akka (or Reactor
> > > > > > https://github.com/reactor/reactor).  Coherence also managed
to
> > get
> > > it
> > > > > > right (in my opinion, that is).
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message