Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <BBDA493B-54B0-4D17-9DC0-DFD7B6F8DBE6@apache.org>
References: <CALkYvqmzCK5VSmE88RobmqOMr4gzKW3Vhww6mS-yUi+A3XogQw@mail.gmail.com>
 <BBDA493B-54B0-4D17-9DC0-DFD7B6F8DBE6@apache.org>
From: Vladimir Ozerov <vozerov@gridgain.com>
Date: Fri, 1 Dec 2017 16:27:35 +0300
Message-ID: <CAGt6o8krSvSW8gjJuHuz0HC3i9zc5i3OpRes7_JgdcGUQKQ+Kw@mail.gmail.com>
Subject: Re: Internal problems requiring graceful node shutdown, reboot, etc.
To: dev@ignite.apache.org
Content-Type: multipart/alternative; boundary="089e0822c7a0f2655b055f475437"
archived-at: Fri, 01 Dec 2017 13:27:41 -0000

--089e0822c7a0f2655b055f475437
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

HI Dmitry,

I do not think it is good idea to mix failures of different threads into a
single event type.
- Practice shows that the most common source of problem is exchange thread
- if disco worker has died, not will be excluded from topology safely
- "nio-acceptor" can be spawn from multiple places where GridNioServer is
started, not all of them are ciritical
- "grid-nio-worker-tcp-comm" is internal thread which doesn't do any
complex processing, so risk of it's crash is minimal

We could track most of them, but death of different threads may result in
different actions from user side. So I propose to start with exchange
thread only for now.

Another important point, is that FailureProcessingPolicy should get enough
information on what happened in order to decide how to react. E.g., as I
explained earlier, IgniteOutOfMemoryException *is not critical error*.
Nasty, but not deadly. And node should not be stopped blindly in response
to this event.

Vladimir.


On Fri, Dec 1, 2017 at 3:50 AM, Denis Magda <dmagda@apache.org> wrote:

> Hi Dmitriy,
>
> I=E2=80=99m totally for the FailureProcessingPolicy addition to
> IgniteConfiguration.
>
> Apart of this, may I ask you to create corresponding documentation ticket=
s
> for 2.4 release and =E2=80=9Cdocumentation=E2=80=9D component? Only for t=
he improvements
> that are getting into the next release. Basically you can aggregate them =
if
> it helps. Feel free to assign the ticket on me right away.
>
> =E2=80=94
> Denis
>
> > On Nov 30, 2017, at 10:31 AM, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=
=B9 =D0=A1=D0=BE=D1=80=D0=BE=D0=BA=D0=B8=D0=BD <sbt.sorokin.dvl@gmail.com>
> wrote:
> >
> > Hi, Igniters!
> >
> > We have a set of internal problems, which required graceful node
> shutdown,
> > or other reaction configured (See discussion thread
> > http://apache-ignite-developers.2346864.n4.nabble.
> com/Ignite-Enhancement-Proposal-7-Internal-problems-detection-td24460.htm=
l
> > ):
> > - IgniteOutOfMemoryException -
> > https://issues.apache.org/jira/browse/IGNITE-6892
> > - Persistence errors - https://issues.apache.org/jira/browse/IGNITE-689=
1
> > - ExchangeWorker exits with error -
> > https://issues.apache.org/jira/browse/IGNITE-6890
> >
> > First, I propose reconsider 3rd problem as "System worker exit while no=
de
> > still running (node stopping process has not been started)", because we
> > have at least 5 worker classes, which running is critical for node
> working.
> >
> > These workers are:
> > - partition-exchanger (ExchangeWorker)
> > - disco-event-worker
> > - nio-acceptor
> > - grid-nio-worker-tcp-comm-*
> > - grid-timeout-worker
> >
> > Second, I propose to use FailureProcessingPolicy (already implemented i=
n
> > scope of task IGNITE-6890) for reaction definition on 1st and 2nd
> detected
> > problems too. This policy can be configured similar to SegmentationPoli=
cy
> > in IgniteConfiguration.
> >
> > Opinions?
>
>

--089e0822c7a0f2655b055f475437--