Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
From: Nikolay Izhikov <nizhikov.dev@gmail.com>
Subject: Re: Monitoring of active transactions
To: dev@ignite.apache.org
References: <CADM=oMYLL6pPtxx-rT+rmOmb6ckygT_qD9VXu-TdQVX38Egw+Q@mail.gmail.com>
Message-ID: <4cabb8f1-95f7-3059-459b-c8453d85e07a@gmail.com>
Date: Fri, 8 Sep 2017 14:57:36 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.2.1
MIME-Version: 1.0
In-Reply-To: <CADM=oMYLL6pPtxx-rT+rmOmb6ckygT_qD9VXu-TdQVX38Egw+Q@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
archived-at: Fri, 08 Sep 2017 11:57:44 -0000

Hello, Ilya.

Great! Thanks!

Can I extend your idea a bit?

I think it be very useful monitor all-user provided listeners and 
callbacks also.

To handle following scenarios:

1а. User starts a job with ExecutorService, IgniteCompute or similar.
1b. User creates ContinuousQuery with remoteFilter and localListener.
2. It take huge amount of time to execute user callback on some node. Or 
thread blocks on some monitor inside callback.

In that case Ignite can detect it and print some waring message.
We can cancel user callback to free resources in some cases.

Specific timeouts and cancel policy should be configured by user.

May be it already covered by FailOverSpi [1] but I can't find 
description of such feature.

We can take WebSphere hangs detection mechanism [2], [3] as an example.

[1] https://apacheignite.readme.io/docs/fault-tolerance
[2] 
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/hung_thread_detection_in_websphere_application_server?lang=en
[3] 
https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.iseries.doc/ae/ttrb_confighangdet.html


08.09.2017 14:27, Ilya Lantukh пишет:
> Igniters,
> 
> According to our current design and implementation, unclosed transaction or
> unreleased lock can hang ignite cluster forever. This is logical, and with
> correct usage of those mechanics such issue should never happen, in real
> world developers can make mistakes and leave transaction open. We have a
> feature "transaction timeout", but turns out it doesn't work in all cases
> (see https://issues.apache.org/jira/browse/IGNITE-6181). Even if all known
> issues are fixed, there is still a lot of room for mistake and incorrect
> usage.
> 
> To make it possible for Ignite users to discover such problem and trace it
> to a particular part of code, I've created a very simple utility that
> collects and prints information about long running transactions for the
> whole cluster. It is available here:
> https://github.com/ilantukh/IgniteTxViewer.
> 
> One might expect such monitoring utilities to be included in Ignite
> codebase. Personally, I think that such information should be available
> from public API, without using of additional applications or diving into
> Ignite internals.
> 
> What do you think?
>