ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Vinogradov <avinogra...@gridgain.com>
Subject Re: Facility to detect long STW pauses and other system response degradations
Date Mon, 20 Nov 2017 14:47:22 GMT
Yakov,

Issue is https://issues.apache.org/jira/browse/IGNITE-6171

We split issue to
#1 STW duration metrics
#2 External monitoring allows to stop node during STW

> Testing GC pause with java thread is
> a bit strange and can give info only after GC pause finishes.

That's ok since it's #1

On Mon, Nov 20, 2017 at 5:45 PM, Dmitriy_Sorokin <sbt.sorokin.dvl@gmail.com>
wrote:

> I have tested solution with java-thread and GC logs had contain same pause
> values of thread stopping which was detected by java-thread.
>
>
> My log (contains pauses > 100ms):
> [2017-11-20 17:33:28,822][WARN ][Thread-1][root] Possible too long STW
> pause: 507 milliseconds.
> [2017-11-20 17:33:34,522][WARN ][Thread-1][root] Possible too long STW
> pause: 5595 milliseconds.
> [2017-11-20 17:33:37,896][WARN ][Thread-1][root] Possible too long STW
> pause: 3262 milliseconds.
> [2017-11-20 17:33:39,714][WARN ][Thread-1][root] Possible too long STW
> pause: 1737 milliseconds.
>
> GC log:
> gridgain@dell-5580-92zc8h2:~$ cat
> ./dev/ignite-logs/gc-2017-11-20_17-33-27.log | grep Total
> 2017-11-20T17:33:27.608+0300: 0,116: Total time for which application
> threads were stopped: 0,0000845 seconds, Stopping threads took: 0,0000246
> seconds
> 2017-11-20T17:33:27.667+0300: 0,175: Total time for which application
> threads were stopped: 0,0001072 seconds, Stopping threads took: 0,0000252
> seconds
> 2017-11-20T17:33:28.822+0300: 1,330: Total time for which application
> threads were stopped: 0,5001082 seconds, Stopping threads took: 0,0000178
> seconds    // GOT!
> 2017-11-20T17:33:34.521+0300: 7,030: Total time for which application
> threads were stopped: 5,5856603 seconds, Stopping threads took: 0,0000229
> seconds    // GOT!
> 2017-11-20T17:33:37.896+0300: 10,405: Total time for which application
> threads were stopped: 3,2595700 seconds, Stopping threads took: 0,0000223
> seconds    // GOT!
> 2017-11-20T17:33:39.714+0300: 12,222: Total time for which application
> threads were stopped: 1,7337123 seconds, Stopping threads took: 0,0000121
> seconds    // GOT!
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message