ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Perfomance drop
Date Wed, 11 Sep 2019 13:08:18 GMT
Hello!

I can see that there's huge unevenness with regards to striped pool usage:
around 80% of load end up on just two striped threads,
sys-stripe-11-#12%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415% and
sys-stripe-9-#10%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415%.
That's 2/3 of all utilization of all theads.

I imagine there is a lot of contention around a very narrow set of keys.
Maybe you update a few keys continuously which will cause locking issues.
Please check your data partitioning and affinity.

Regards,
-- 
Ilya Kasnacheev


чт, 5 сент. 2019 г. в 18:12, Кузин Никита (Nikita Kuzin) <
nkuzin@iitdgroup.ru>:

> jfr from ignite is in attachment
>
> _________________________________
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nkuzin@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
> ------------------------------
> *От:* Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
> *Отправлено:* 2 сентября 2019 г. 14:08
> *Кому:* user@ignite.apache.org <user@ignite.apache.org>
> *Тема:* Re: Perfomance drop
>
> Hello!
>
> If GC is fine next step is examining thread dumps and logs when a long
> operation is taking place. You can trigger it yourself by calling
> IgniteUtils.dumpThreads() in a test setting.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 2 сент. 2019 г. в 11:11, Кузин Никита (Nikita Kuzin) <
> nkuzin@iitdgroup.ru>:
>
> Hello!
>
> gc log for 3 days of working ignite.
>
> There are 58 GCs with duration 100-200ms and 9 GCs with duration of
> 200-300ms
>
> _________________________________
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nkuzin@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
> ------------------------------
> *От:* Кузин Никита (Nikita Kuzin) <nkuzin@iitdgroup.ru>
> *Отправлено:* 31 августа 2019 г. 10:26
> *Кому:* user@ignite.apache.org <user@ignite.apache.org>
> *Тема:* RE: Perfomance drop
>
> Hello!
>
> We collected GC log, it seems to be ok.
>
> Long duration operations are very random events, we will try to collect.
>
> Can the caches' eviction processes has influence on cache reading? (use of
> same threads)
> _________________________________
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nkuzin@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
> ------------------------------
> *От:* Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
> *Отправлено:* 30 августа 2019 г. 15:49
> *Кому:* user@ignite.apache.org <user@ignite.apache.org>
> *Тема:* Re: Perfomance drop
>
> Hello!
>
> Have you tried collecting thread dumps/JFR/etc while such an operation is
> waiting? It could provide some clues.
>
> Do you have GC log collected? Any long GC pauses?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 30 авг. 2019 г. в 14:43, Кузин Никита (Nikita Kuzin) <
> nkuzin@iitdgroup.ru>:
>
> Hi!
>
> We have application that goes to ignite node (only one node in grid).
> Application and ignite are located on one machine.
>
> And we noticed that sometimes that cache located at ignite is very slow:
> we execute method igniteCache.containsKey(key) and it can lasts 900ms, size
> of cache less than 10000 entries
> This cache is used mostly for read.
>
> Persistence is off
> All of JVM recommendations is applied
> OS: windows server 2012 R2
>
> _________________________________
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nkuzin@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
>
>
Mime
View raw message