ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Gura <ag...@apache.org>
Subject Re: GridDhtInvalidPartitionException takes the cluster down
Date Wed, 27 Mar 2019 15:24:50 GMT
What do you think about including patches [1] and [2] to Ignite 2.7.5?
It's all about default failure handler behavior in cases of
SYSTEM_WORKER_BLOCKED and SYSTEM_CRITICAL_OPERATION_TIMEOUT.

Andrey Kuznetsov, could you please check, does IGNITE-10003 depend on
other issue that isn't included into 2.7 release?

[1] https://issues.apache.org/jira/browse/IGNITE-10154
[2] https://issues.apache.org/jira/browse/IGNITE-10003

On Wed, Mar 27, 2019 at 8:11 AM Denis Magda <dmagda@apache.org> wrote:
>
> Folks, thanks for sharing details and inputs. This is helpful. As long as I
> spend a lot of time working with Ignite users, I'll look into this topic in
> a couple of days to propose some changes. In the meantime, here is a fresh
> one report on the user list:
> http://apache-ignite-users.70518.x6.nabble.com/Triggering-Rebalancing-Programmatically-get-error-while-requesting-td27651.html
>
>
> -
> Denis
>
>
> On Tue, Mar 26, 2019 at 9:04 AM Andrey Gura <agura@apache.org> wrote:
>
> > CleanupWorker termination can lead to the following effects:
> >
> > - Queries can retrieve data that have to expired so application will
> > behave incorrectly.
> > - Memory and/or disc can be overflowed because entries weren't expired.
> > - Performance degradation is possible due to unmanageable data set grows.
> >
> > On Tue, Mar 26, 2019 at 4:58 PM Roman Shtykh <rshtykh@yahoo.com.invalid>
> > wrote:
> > >
> > > Vyacheslav, if you are talking about this particular case I described, I
> > believe it has no influence on PME. What could happen is having
> > CleanupWorker thread dead (which is not good too).But I believe we are
> > talking in a wider scope.
> > >
> > > -- Roman
> > >
> > >
> > >     On Tuesday, March 26, 2019, 10:23:30 p.m. GMT+9, Vyacheslav Daradur <
> > daradurvs@gmail.com> wrote:
> > >
> > >  In general I agree with Andrey, the handler is very usefull itself. It
> > > allows us to become know that ‘GridDhtInvalidPartitionException’ is not
> > > processed properly in PME process by worker.
> > >
> > > Nikolay, look at the code, if Failure Handler hadles an exception - this
> > > means that while-true loop in worker’s body has been interrupted with
> > > unexpected exception and thread is completed his lifecycle.
> > >
> > > Without Failure Hanller, in the current case, the cluster will hang,
> > > because of unable to participate in PME process.
> > >
> > > So, the problem is the incorrect handling of the exception in PME’s task
> > > wich should be fixed.
> > >
> > >
> > > вт, 26 марта 2019 г. в 14:24, Andrey Kuznetsov <stkuzma@gmail.com>:
> > >
> > > > Nikolay,
> > > >
> > > > Feel free to suggest better error messages to indicate
> > internal/critical
> > > > failures. User actions in response to critical failures are rather
> > limited:
> > > > mail to user-list or maybe file an issue. As for repetitive warnings,
> > it
> > > > makes sense, but requires additional stuff to deliver such signals,
> > mere
> > > > spamming to log will not have an effect.
> > > >
> > > > Anyway, when experienced committers suggest to disable failure
> > handling and
> > > > hide existing issues, I feel as if they are pulling my leg.
> > > >
> > > > Best regards,
> > > > Andrey Kuznetsov.
> > > >
> > > > вт, 26 марта 2019, 13:30 Nikolay Izhikov nizhikov@apache.org:
> > > >
> > > > > Andrey.
> > > > >
> > > > > >  the thread can be made non-critical, and we can restart it
every
> > time
> > > > it
> > > > > dies
> > > > >
> > > > > Why we can't restart critical thread?
> > > > > What is the root difference between critical and non critical
> > threads?
> > > > >
> > > > > > It's much simpler to catch and handle all exceptions in critical
> > > > threads
> > > > >
> > > > > I don't agree with you.
> > > > > We develop Ignite not because it simple!
> > > > > We must spend extra time to made it robust and resilient to the
> > failures.
> > > > >
> > > > > > Failure handling is a last-chance tool that reveals internal
Ignite
> > > > > errors
> > > > > > 100% agree with you: overcome, but not hide.
> > > > >
> > > > > Logging stack trace with proper explanation is not hiding.
> > > > > Killing nodes and whole cluster is not "handling".
> > > > >
> > > > > > As far as I see from user-list messages, our users are qualified
> > enough
> > > > > to provide necessary information from their cluster-wide logs.
> > > > >
> > > > > We shouldn't develop our product only for users who are able to read
> > > > Ignite
> > > > > sources to decrypt the fail reason behind "starvation in stripped
> > pool"
> > > > >
> > > > > Some of my questions remain unanswered :) :
> > > > >
> > > > > 1. How user can know it's an Ignite bug? Where this bug should be
> > > > reported?
> > > > > 2. Do we log it somewhere?
> > > > > 3. Do we warn user before shutdown several times?
> > > > > 4. "starvation in stripped pool" I think it's not clear error
> > message.
> > > > > Let's make it more specific!
> > > > > 5. Let's write to the user log - what he or she should do to prevent
> > this
> > > > > error in future?
> > > > >
> > > > >
> > > > > вт, 26 мар. 2019 г. в 12:13, Andrey Kuznetsov <stkuzma@gmail.com>:
> > > > >
> > > > > > Nikolay,
> > > > > >
> > > > > > >  Why we can't restart some thread?
> > > > > > Technically, we can. It's just matter of design: the thread
can be
> > made
> > > > > > non-critical, and we can restart it every time it dies. But
such
> > design
> > > > > > looks poor to me. It's much simpler to catch and handle all
> > exceptions
> > > > in
> > > > > > critical threads. Failure handling is a last-chance tool that
> > reveals
> > > > > > internal Ignite errors. It's not pleasant for us when users
see
> > these
> > > > > > errors, but it's better than hiding.
> > > > > >
> > > > > > >  Actually, distributed systems are designed to overcome
some
> > bugs,
> > > > > thread
> > > > > > failure, node failure, for example, isn't it?
> > > > > > 100% agree with you: overcome, but not hide.
> > > > > >
> > > > > > >  How user can know it's a bug? Where this bug should be
reported?
> > > > > > As far as I see from user-list messages, our users are qualified
> > enough
> > > > > to
> > > > > > provide necessary information from their cluster-wide logs.
> > > > > >
> > > > > >
> > > > > > вт, 26 мар. 2019 г. в 11:19, Nikolay Izhikov <nizhikov@apache.org
> > >:
> > > > > >
> > > > > > > Andrey.
> > > > > > >
> > > > > > > > As for SYSTEM_WORKER_TERMINATION, it's unrecoverable,
there is
> > no
> > > > use
> > > > > > to
> > > > > > > wait for dead thread's magical resurrection.
> > > > > > >
> > > > > > > Why is it unrecoverable?
> > > > > > > Why we can't restart some thread?
> > > > > > > Is there some kind of nature limitation to not restart
system
> > thread?
> > > > > > >
> > > > > > > Actually, distributed systems are designed to overcome
some bugs,
> > > > > thread
> > > > > > > failure, node failure, for example, isn't it?
> > > > > > > > if under some circumstances node> stop leads to
cascade cluster
> > > > > crash,
> > > > > > > then it's a bug
> > > > > > >
> > > > > > > How user can know it's a bug? Where this bug should be
reported?
> > > > > > > Do we log it somewhere?
> > > > > > > Do we warn user before shutdown one or several times?
> > > > > > >
> > > > > > > This feature kills user experience literally now.
> > > > > > >
> > > > > > > If I would be a user of the product that just shutdown
with poor
> > log
> > > > I
> > > > > > > would throw this product away.
> > > > > > > Do we want it for Ignite?
> > > > > > >
> > > > > > > From SO discussion I see following error message: ": >>>
Possible
> > > > > > > starvation in striped pool."
> > > > > > > Are you sure this message are clear for Ignite user(not
Ignite
> > > > hacker)?
> > > > > > > What user should do to prevent this error in future?
> > > > > > >
> > > > > > > В Вт, 26/03/2019 в 10:10 +0300, Andrey Kuznetsov пишет:
> > > > > > > > By default, SYSTEM_WORKER_BLOCKED failure type is
not handled.
> > I
> > > > > don't
> > > > > > > like
> > > > > > > > this behavior, but it may be useful sometimes: "frozen"
threads
> > > > have
> > > > > a
> > > > > > > > chance to become active again after load decreases.
As for
> > > > > > > > SYSTEM_WORKER_TERMINATION, it's unrecoverable, there
is no use
> > to
> > > > > wait
> > > > > > > for
> > > > > > > > dead thread's magical resurrection. Then, if under
some
> > > > circumstances
> > > > > > > node
> > > > > > > > stop leads to cascade cluster crash, then it's a bug,
and it
> > should
> > > > > be
> > > > > > > > fixed. Once and for all. Instead of hiding the flaw
we have in
> > the
> > > > > > > product.
> > > > > > > >
> > > > > > > > вт, 26 мар. 2019 г. в 09:17, Roman Shtykh
> > > > <rshtykh@yahoo.com.invalid
> > > > > >:
> > > > > > > >
> > > > > > > > > + 1 for having the default settings revisited.
> > > > > > > > > I understand Andrey's reasonings, but sometimes
taking nodes
> > down
> > > > > is
> > > > > > > too
> > > > > > > > > radical (as in my case it was
> > GridDhtInvalidPartitionException
> > > > > which
> > > > > > > could
> > > > > > > > > be ignored for a while when rebalancing <-
I might be wrong
> > > > here).
> > > > > > > > >
> > > > > > > > > -- Roman
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >    On Tuesday, March 26, 2019, 2:52:14 p.m. GMT+9,
Denis
> > Magda <
> > > > > > > > > dmagda@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > p    Nikolay,
> > > > > > > > > Thanks for kicking off this discussion. Surprisingly,
> > planned to
> > > > > > start
> > > > > > > a
> > > > > > > > > similar one today and incidentally came across
this thread.
> > > > > > > > > Agree that the failure handler should be off
by default or
> > the
> > > > > > default
> > > > > > > > > settings have to be revisited. That's true that
people are
> > > > > > complaining
> > > > > > > of
> > > > > > > > > nodes shutdowns even on moderate workloads. For
instance,
> > that's
> > > > > the
> > > > > > > most
> > > > > > > > > recent feedback related to slow checkpointing:
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://stackoverflow.com/questions/55299337/stripped-pool-starvation-in-wal-writing-causes-node-cluster-node-failure
> > > > > > > > >
> > > > > > > > > At a minimum, let's consider the following:
> > > > > > > > >    - A failure handler needs to provide hints
on how to come
> > > > around
> > > > > > the
> > > > > > > > > shutdown in the future. Take the checkpointing
SO thread
> > above.
> > > > > It's
> > > > > > > > > unclear from the logs how to prevent the same
situation next
> > time
> > > > > > > (suggest
> > > > > > > > > parameters for tuning, flash drives, etc).
> > > > > > > > >    - Is there any protection for a full cluster
restart? We
> > need
> > > > to
> > > > > > > > > distinguish a slow cluster from the stuck one.
A node removal
> > > > > should
> > > > > > > not
> > > > > > > > > lead to a meltdown of the whole storage.
> > > > > > > > >    - Should we enable the failure handler for
things like
> > > > > > transactions
> > > > > > > or
> > > > > > > > > PME and have it off for checkpointing and something
else?
> > Let's
> > > > > have
> > > > > > it
> > > > > > > > > enabled for cases when we are 100% certain that
a node
> > shutdown
> > > > is
> > > > > > the
> > > > > > > > > right thing and print out warnings with suggestions
whenever
> > > > we're
> > > > > > not
> > > > > > > > > confident that the removal is appropriate.
> > > > > > > > > --Denis
> > > > > > > > >
> > > > > > > > > On Mon, Mar 25, 2019 at 5:52 AM Andrey Gura <
> > agura@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Failure handlers were introduced in order to
avoid cluster
> > > > hanging
> > > > > > and
> > > > > > > > > they kill nodes instead.
> > > > > > > > >
> > > > > > > > > If critical worker was terminated by
> > > > > GridDhtInvalidPartitionException
> > > > > > > > > then your node is unable to work anymore.
> > > > > > > > >
> > > > > > > > > Unexpected cluster shutdown with reasons in logs
that failure
> > > > > > handlers
> > > > > > > > > provide is better than hanging. So answer is
NO. We mustn't
> > > > disable
> > > > > > > > > failure handlers.
> > > > > > > > >
> > > > > > > > > On Mon, Mar 25, 2019 at 2:47 PM Roman Shtykh
> > > > > > <rshtykh@yahoo.com.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > If it sticks to the behavior we had before
introducing
> > failure
> > > > > > > handler,
> > > > > > > > >
> > > > > > > > > I think it's better to have disabled instead
of killing the
> > whole
> > > > > > > cluster,
> > > > > > > > > as in my case, and create a parent issue for
those ten
> > > > bugs.Pavel,
> > > > > > > thanks
> > > > > > > > > for the suggestion!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >    On Monday, March 25, 2019, 7:07:20 p.m.
GMT+9, Nikolay
> > > > > Izhikov
> > > > > > <
> > > > > > > > >
> > > > > > > > > nizhikov@apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > >  Guys.
> > > > > > > > > >
> > > > > > > > > > We should fix the SYSTEM_WORKER_TERMINATION
once and for
> > all.
> > > > > > > > > > Seems, we have ten or more "cluster shutdown"
bugs with
> > this
> > > > > > > subsystem
> > > > > > > > > > since it was introduced.
> > > > > > > > > >
> > > > > > > > > > Should we disable it by default in 2.7.5?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > пн, 25 мар. 2019 г. в 13:04, Pavel
Kovalenko <
> > > > jokserfn@gmail.com
> > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi Roman,
> > > > > > > > > > >
> > > > > > > > > > > I think this InvalidPartition case
can be simply handled
> > > > > > > > > > > in GridCacheTtlManager.expire method.
> > > > > > > > > > > For workaround a custom FailureHandler
can be configured
> > that
> > > > > > will
> > > > > > > not
> > > > > > > > >
> > > > > > > > > stop
> > > > > > > > > > > a node in case of such exception is
thrown.
> > > > > > > > > > >
> > > > > > > > > > > пн, 25 мар. 2019 г. в 08:38,
Roman Shtykh
> > > > > > > <rshtykh@yahoo.com.invalid>:
> > > > > > > > > > >
> > > > > > > > > > > > Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > Restarting a node when injecting
data and having it
> > > > expired,
> > > > > > > results
> > > > > > > > >
> > > > > > > > > at
> > > > > > > > > > > > GridDhtInvalidPartitionException
which terminates nodes
> > > > with
> > > > > > > > > > > > SYSTEM_WORKER_TERMINATION one
by one taking the whole
> > > > cluster
> > > > > > > down.
> > > > > > > > >
> > > > > > > > > This
> > > > > > > > > > > is
> > > > > > > > > > > > really bad and I didn't find the
way to save the
> > cluster
> > > > from
> > > > > > > > > > >
> > > > > > > > > > > disappearing.
> > > > > > > > > > > > I created a JIRA issue
> > > > > > > > > > >
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-11620
> > > > > > > > > > > > with a test case. Any clues how
to fix this
> > inconsistency
> > > > > when
> > > > > > > > > > >
> > > > > > > > > > > rebalancing?
> > > > > > > > > > > >
> > > > > > > > > > > > -- Roman
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > >  Andrey Kuznetsov.
> > > > > >
> > > > >
> > > >
> > > --
> > > Best Regards, Vyacheslav D.
> >

Mime
View raw message