ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Ignite Enhancement Proposal #7 (Internal problems detection)
Date Tue, 21 Nov 2017 17:25:55 GMT
A lack of suggestions and thoughts encourages me to create a ticket:
https://issues.apache.org/jira/browse/IGNITE-6980 <https://issues.apache.org/jira/browse/IGNITE-6980>

—
Denis

> On Nov 20, 2017, at 2:53 PM, Denis Magda <dmagda@apache.org> wrote:
> 
> If an Ignite operation hangs by some reason due to an internal problem or buggy application
code it needs to eventual *time out*.
> 
> Take atomic operations case brought by Val to our attention recently:
> http://apache-ignite-developers.2346864.n4.nabble.com/Timeouts-in-atomic-cache-td19839.html
> 
> An application must not freeze waiting for a human being intervention if an atomic update
fails internally.
> 
> Even more I would let all possible operation to time out:
> - Ignite compute computations.
> - Ignite services calls.
> - Atomic/transactional cache updates.
> - SQL queries.
> 
> I’m not sure this is covered by any of the tickets from the IEP-7. Any thoughts/suggestion
before the one is created?
> 
> —
> Denis
> 
>> On Nov 20, 2017, at 8:56 AM, Anton Vinogradov <avinogradov@gridgain.com> wrote:
>> 
>> Dmitry,
>> 
>> There's two cases
>> 1) STW duration is long -> notifying monitoring via JMX metric
>> 
>> 2) STW duration exceed N seconds -> no need to wait for something.
>> We already know that node will be segmented or that pause bigger that N
>> seconds will affect cluster performance.
>> Better case is to kill node ASAP to protect the cluster. Some customers
>> have huge timeouts and such node can kill whole cluster in case it will not
>> be killed by watchdog.
>> 
>> On Mon, Nov 20, 2017 at 7:23 PM, Dmitry Pavlov <dpavlov.spb@gmail.com>
>> wrote:
>> 
>>> Hi Anton,
>>> 
>>>> - GC STW duration exceed maximum possible length (node should be stopped
>>> before
>>> STW finished)
>>> 
>>> Are you sure we should kill node in case long STW? Can we produce warnings
>>> into logs and monitoring tools an wait node to become alive a little bit
>>> longer if we detect STW. In this case we can notify coordinator or other
>>> node, that 'current node is in STW, please wait longer than 3 heartbeat
>>> timeout'.
>>> 
>>> It is probable such pauses will occur not often?
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> пн, 20 нояб. 2017 г. в 18:53, Anton Vinogradov <avinogradov@gridgain.com>:
>>> 
>>>> Igniters,
>>>> 
>>>> Internal problems may and, unfortunately, cause unexpected cluster
>>>> behavior.
>>>> We should determine behavior in case any of internal problem happened.
>>>> 
>>>> Well known internal problems can be split to:
>>>> 1) OOM or any other reason cause node crash
>>>> 
>>>> 2) Situations required graceful node shutdown with custom notification
>>>> - IgniteOutOfMemoryException
>>>> - Persistence errors
>>>> - ExchangeWorker exits with error
>>>> 
>>>> 3) Prefomance issues should be covered by metrics
>>>> - GC STW duration
>>>> - Timed out tasks and jobs
>>>> - TX deadlock
>>>> - Hanged Tx (waits for some service)
>>>> - Java Deadlocks
>>>> 
>>>> I created special issue [1] to make sure all these metrics will be
>>>> presented at WebConsole or VisorConsole (what's preferred?)
>>>> 
>>>> 4) Situations required external monitoring implementation
>>>> - GC STW duration exceed maximum possible length (node should be stopped
>>>> before STW finished)
>>>> 
>>>> All this problems were reported by different persons different time ago,
>>>> So, we should reanalyze each of them and, possible, find better ways to
>>>> solve them than it described at issues.
>>>> 
>>>> P.s. IEP-7 [2] already contains 9 issues, feel free to mention something
>>>> else :)
>>>> 
>>>> [1] https://issues.apache.org/jira/browse/IGNITE-6961
>>>> [2]
>>>> 
>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
>>> 7%3A+Ignite+internal+problems+detection
>>>> 
>>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message