Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <DM2PR0701MB1230B00973120AF3E45CEE84CFAB0@DM2PR0701MB1230.namprd07.prod.outlook.com>
References: <CAGt6o8mTZqhd1SzwnzY4yy8PD-YaVK3RScHv0=o+qbwF2QXhjw@mail.gmail.com>
 <CA+eZwrHffDk=E-F170cPOXZpZHn0=qanmxMF3uYCzoe-9mR3aQ@mail.gmail.com>
 <CA+0=VoVk0ar3m_Y3+d9-Q0F0gi4CsSoJqd4JNSgSgg2aAcaSHA@mail.gmail.com>
 <dba8b535-d70c-4f9b-a5ef-3258dd0d02ea@misys.com> <CAGt6o8m4tAbTXLCS389cR_MGdysrcVondLyGNhJ8m_eg4jYMKg@mail.gmail.com>
 <CAGt6o8nUaMQ9VXdTrCDxkpoNZW4xMh6xJrZxWpnE6ff7EmeH4A@mail.gmail.com>
 <CAGt6o8kV=KgwMrPiSVcBie0Ugio8gA7=NZYALR6j=Z8zZJ1UCw@mail.gmail.com>
 <CAMbfdjbAvGOdWQ-88D5-gT94rh+Saaa0ms5R1qWb4suav-dz_w@mail.gmail.com>
 <CA+0=VoVv2Xy1i+kE=TXAdisn2mWW=v6acw2i=-iVt-C2BFpqfg@mail.gmail.com>
 <CAGcMBHiqSWxcnVGhoXWW8T48Kn2KVr0ZFC+U8eV8bVrWsyJA4w@mail.gmail.com>
 <CA+0=VoXPzNnZmEwMdUw3RKeBfeu0P7s8gmuGb-8rFm64dKewnQ@mail.gmail.com>
 <DM2PR0701MB12300C1DA65C92C0D66C6E65CFAB0@DM2PR0701MB1230.namprd07.prod.outlook.com>
 <CAGt6o8mBN4rett1SHPXi5NbqrzHXAq8V0dD6VTRkV=POkkgxKg@mail.gmail.com> <DM2PR0701MB1230B00973120AF3E45CEE84CFAB0@DM2PR0701MB1230.namprd07.prod.outlook.com>
From: Vladimir Ozerov <vozerov@gridgain.com>
Date: Thu, 27 Oct 2016 14:21:30 +0300
Message-ID: <CAGt6o8nh0pHGfOHFedvOZw=CXwbQWfR2LJKfA1JMFMOXxiHbiQ@mail.gmail.com>
Subject: Re: Create separate thread pool for query execution
To: dev@ignite.apache.org
Content-Type: multipart/alternative; boundary=001a114358127de59f053fd6f15f
archived-at: Thu, 27 Oct 2016 11:21:41 -0000

--001a114358127de59f053fd6f15f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Andrey,

I am not aware of any serious issues with thread pools configuration. For
cache pool ("system pool") [cores * 2] makes sense, because usually cache
operations are fast and small, but contains blocking operations around
(e.g. moving task from blocking queue to the thread). Sometimes [cores * 2]
could be too much for public pool, but again - I do not know any serious
problems with it except of slight performance degradation.

The main idea of many thread pool is to allow users to perform synchronous
operations to one component from within another. For sure, there is a
general solution to the problem - reactive style or continuations. When
every potentially blocking operation releases current thread and allow it
to perform other stuff. Something like ComputeJobContinuation.holdcc/callcc=
,
but performed implicitly. In this case it would be enough to have a single
thread pool for everything. The problem is that this approach is too
complex for both users and Ignite developers. This is the main reason why
we have multiple pools - essentially we create separate execution circuits
for different components, thus allowing them interact with each other in
blocking fashion. Personally, I am completely satisfied with this approach,
because it is simple for users.

What important here, is that we are not trying to support each and every
use case. Instead, we are trying to cover some widely-used scenarios, such
as "call cache from compute".

Queries cannot be run in public pool, because tasks and closures are also
executed in that pool. It could lead to starvation when query is started
from compute job/closure.

Vladimir.


On Thu, Oct 27, 2016 at 4:40 AM, Andrey Kornev <andrewkornev@hotmail.com>
wrote:

> Vladimir,
>
>
> Thanks for your reply!
>
>
> Unfortunately, most of the time users end up having to think about
> configuring the thread pools, because the "sensible" default value for mo=
st
> of the thread pools is the number of processors multiplied by 2. With
> modern production-grade multicore machines, the default thread pools size
> is simply too high.
>
>
> Another problem with the current approach to thread management is that
> some resources inevitably go wasted. For example, if one thread pool runs
> out of threads it won't be able to borrow from another thread pool even
> though that one may have plenty of threads sitting idle.
>
>
> And even if there were really good reasons for maintaining all those
> thread pools, complete lack of any documentation makes it pretty difficul=
t
> (if not impossible) to configure the thread pools correctly.
>
>
> Finally, your explanation as to why another thread pool - dedicated
> exclusively to queries - is necessary, doesn't explain anything, but rath=
er
> raises more questions:
>
> - why would a query block cache operations? Even if it absolutely must
> block, so what? In this scenario, how is a query different from another
> "regular" cache operation that may also block some other cache operations=
?
> Why special treatment?
>
> - why wouldn't it possible to run queries from the Ignite closures and
> tasks? I don't see any connection.
>
>
> Thanks
>
> Andrey
>
> ________________________________
> From: Vladimir Ozerov <vozerov@gridgain.com>
> Sent: Tuesday, October 25, 2016 11:40 PM
> To: dev@ignite.apache.org
> Subject: Re: Create separate thread pool for query execution
>
> Andrey,
>
> Ignite already works the way you described except of listeners/callbacks.
> Only size of each thread pool is exposed to configuration. And each one
> already have sensible default value, so normally users do not think of it
> at all. For some pools we also have timeouts, but they will be deprecated
> soon.
>
> Queries cannot be executed in system pool because they can block cache
> operations. And they cannot be executed in public pool because in this ca=
se
> it will be impossible to run queries from closures and tasks.
>
> Vladimir.
>
> On Wed, Oct 26, 2016 at 8:37 AM, Andrey Kornev <andrewkornev@hotmail.com>
> wrote:
>
> > Guys,
> > I feel very uneasy about proliferation of thread pools in Ignite. I
> > suggest we take step back.
> > Most of the users (including yours truly) have no idea how to correctly
> > configure the existing thread pools, what those thread pools are for
> (like,
> > management thread pool, or marshaller cache thread pool, or async
> callback
> > thread pool, or peer class loading thread pool, or - my personal
> favorite -
> > utility cache thread pool, just to name a few), and why they should wor=
ry
> > about them altogether.
> > In reality, there should probably be only 2 parameters exposed at the
> > configuration level: the size of Ignite's internal (or "system") thread
> > pool, and the size of the user thread pool. Or alternatively, one numbe=
r
> > could indicate the total number of thread available to Ignite (hard upp=
er
> > bound) and the other would be a ratio of system threads to user threads=
.
> > Either way, it's Ignite's internal business how to manage the thread
> pools
> > given the constraints. For example, Ignite may choose to allocate the
> > system threads across twenty thread pools, or maybe just one -- the use=
rs
> > do not care. What users do care about is the size of the user thread po=
ol
> > because it's the one that executes their code. I know it's not the case
> in
> > the current version, but going forward, Ignite must ensure that all use=
r
> > code (including all sorts of listeners and callbacks) is executed on th=
e
> > user threads.
> > In any case, it's the user who is ultimately in charge of figuring out
> the
> > correct thread pool sizes, and I believe having just 2 knobs to turn vs=
.
> 7
> > or 8 as is the case today, would make things a lot simpler and reduce t=
he
> > guess work.
> > Speaking of query execution, it feels natural to have it executed on th=
e
> > user thread pool, as the query is a user-triggered action executing a
> > user's code (even though the code is not Java in this case, but somethi=
ng
> > that looks rather like SQL).
> > Regards,
> > Andrey
> > _____________________________
> > From: Dmitriy Setrakyan <dsetrakyan@apache.org<mailto:
> > dsetrakyan@apache.org>>
> > Sent: Monday, October 24, 2016 9:23 AM
> > Subject: Re: Create separate thread pool for query execution
> > To: <dev@ignite.apache.org<mailto:dev@ignite.apache.org>>
> >
> >
> > I think this makes sense. Do we have a ticket filed?
> >
> > On Mon, Oct 24, 2016 at 2:17 AM, Yakov Zhdanov <yzhdanov@apache.org
> > <mailto:yzhdanov@apache.org>> wrote:
> >
> > > > As far as the query thread pool, how many threads should be in it b=
y
> > > > default? What happens is the query is run from compute task - will
> the
> > > > execution be shifted from the public to the query thread pool?
> > >
> > > Pool management should be the same as for other pools.
> > >
> > > Remote executions should be executed in query pool. Local should run
> > > synchronously.
> > >
> > > --Yakov
> > >
> > > 2016-10-24 11:53 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
> > <mailto:dsetrakyan@apache.org>>:
> > >
> > > > Sergey, I think we already have a general approach with 3 or 4 thre=
ad
> > > pools
> > > > we have today.
> > > >
> > > > As far as the query thread pool, how many threads should be in it b=
y
> > > > default? What happens is the query is run from compute task - will
> the
> > > > execution be shifted from the public to the query thread pool?
> > > >
> > > > D.
> > > >
> > > > On Mon, Oct 24, 2016 at 1:47 AM, Sergey Kozlov <skozlov@gridgain.co=
m
> > <mailto:skozlov@gridgain.com>>
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > It seems we need a set of dedicated pools for various purposes.
> Could
> > > we
> > > > > design a general apporach for this like following:
> > > > > - define dedicated pools in the grid configuration
> > > > > - run a query/compute/etc in the particular pool
> > > > >
> > > > > On Mon, Oct 24, 2016 at 9:49 AM, Vladimir Ozerov <
> > vozerov@gridgain.com<mailto:vozerov@gridgain.com>
> > > >
> > > > > wrote:
> > > > >
> > > > > > Romain,
> > > > > > We do not pre-start threads in our pools on Ignite start.
> Moreover,
> > > > idle
> > > > > > threads are stopped after some timeout.
> > > > > >
> > > > > > 24 =D0=BE=D0=BA=D1=82. 2016 =D0=B3. 8:57 =D0=BF=D0=BE=D0=BB=D1=
=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C "Gilles, Romain" <
> > > > > > romain.gilles@misys.com<mailto:romain.gilles@misys.com>>
> > > > > > =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB:
> > > > > >
> > > > > > Hi igniters,
> > > > > > I'm not against to create a thread pool for each features but I
> > have
> > > > some
> > > > > > difficulty to minimize the number of threads required to start
> > > > ignite...
> > > > > If
> > > > > > you add too many thread pools does this will not increase the
> > number
> > > of
> > > > > > threads at startup?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Romain
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > De: Dmitriy Setrakyan <dsetrakyan@apache.org<mailto:
> > dsetrakyan@apache.org>>
> > > > > > Envoye: 23 oct. 2016 23:00
> > > > > > A: dev@ignite.apache.org<mailto:dev@ignite.apache.org>
> > > > > > Objet: Re: Create separate thread pool for query execution
> > > > > >
> > > > > > How about long running compute? Do we need a separate thread po=
ol
> > for
> > > > it
> > > > > as
> > > > > > well?
> > > > > >
> > > > > > On Sun, Oct 23, 2016 at 3:57 AM, Sergi Vladykin <
> > > > > sergi.vladykin@gmail.com<mailto:sergi.vladykin@gmail.com>>
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Sergi
> > > > > > >
> > > > > > > 2016-10-23 11:52 GMT+03:00 Vladimir Ozerov <
> vozerov@gridgain.com
> > <mailto:vozerov@gridgain.com>>:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > Currently if several long-running queries are submitted, it
> may
> > > > stall
> > > > > > all
> > > > > > > > cache operations because all theads in the system pool will
> be
> > > busy
> > > > > > with
> > > > > > > > queries.
> > > > > > > >
> > > > > > > > I think it makes sense to move queries execution into
> separate
> > > > > > dedicated
> > > > > > > > thread pool. As we have timeouts for core pool threads now,
> it
> > > > should
> > > > > > not
> > > > > > > > affect memory consumption or startup time anyhow. Thoughts?
> > > > > > > >
> > > > > > > > Vladimir.
> > > > > > > >
> > > > > > >
> > > > > > "Misys" is the trade name of the Misys group of companies. This
> > email
> > > > and
> > > > > > any attachments have been scanned for known viruses using
> multiple
> > > > > > scanners. This email message is intended for the named recipien=
t
> > > only.
> > > > It
> > > > > > may be privileged and/or confidential. If you are not the named
> > > > recipient
> > > > > > of this email please notify us immediately and do not copy it o=
r
> > use
> > > it
> > > > > for
> > > > > > any purpose, nor disclose its contents to any other person. Thi=
s
> > > email
> > > > > does
> > > > > > not constitute the commencement of legal relations between you
> and
> > > > Misys.
> > > > > > Please refer to the executed contract between you and the
> relevant
> > > > member
> > > > > > of the Misys group for the identity of the contracting party wi=
th
> > > which
> > > > > you
> > > > > > are dealing.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sergey Kozlov
> > > > > GridGain Systems
> > > > > www.gridgain.com<http://www.gridgain.com>
> > > > >
> > > >
> > >
> >
> >
> >
>

--001a114358127de59f053fd6f15f--