ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Кузин Никита (Nikita Kuzin) <nku...@iitdgroup.ru>
Subject Re: Perfomance drop
Date Wed, 11 Sep 2019 17:57:32 GMT
Hello!

How we can debug it to find cache name that causes it?
Some extra logging or metrics?

Скачайте Outlook для Android<https://aka.ms/ghei36>

________________________________
From: Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
Sent: Wednesday, September 11, 2019 4:08:18 PM
To: user@ignite.apache.org <user@ignite.apache.org>
Subject: Re: Perfomance drop

Hello!

I can see that there's huge unevenness with regards to striped pool usage: around 80% of load
end up on just two striped threads,
sys-stripe-11-#12%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415% and
sys-stripe-9-#10%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415%. That's 2/3 of all
utilization of all theads.

I imagine there is a lot of contention around a very narrow set of keys. Maybe you update
a few keys continuously which will cause locking issues. Please check your data partitioning
and affinity.

Regards,
--
Ilya Kasnacheev


чт, 5 сент. 2019 г. в 18:12, Кузин Никита (Nikita Kuzin) <nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>>:
jfr from ignite is in attachment

_________________________________
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27
________________________________
От: Ilya Kasnacheev <ilya.kasnacheev@gmail.com<mailto:ilya.kasnacheev@gmail.com>>
Отправлено: 2 сентября 2019 г. 14:08
Кому: user@ignite.apache.org<mailto:user@ignite.apache.org> <user@ignite.apache.org<mailto:user@ignite.apache.org>>
Тема: Re: Perfomance drop

Hello!

If GC is fine next step is examining thread dumps and logs when a long operation is taking
place. You can trigger it yourself by calling IgniteUtils.dumpThreads() in a test setting.

Regards,
--
Ilya Kasnacheev


пн, 2 сент. 2019 г. в 11:11, Кузин Никита (Nikita Kuzin) <nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>>:
Hello!

gc log for 3 days of working ignite.

There are 58 GCs with duration 100-200ms and 9 GCs with duration of 200-300ms

_________________________________
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27
________________________________
От: Кузин Никита (Nikita Kuzin) <nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>>
Отправлено: 31 августа 2019 г. 10:26
Кому: user@ignite.apache.org<mailto:user@ignite.apache.org> <user@ignite.apache.org<mailto:user@ignite.apache.org>>
Тема: RE: Perfomance drop

Hello!

We collected GC log, it seems to be ok.

Long duration operations are very random events, we will try to collect.

Can the caches' eviction processes has influence on cache reading? (use of same threads)
_________________________________
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27
________________________________
От: Ilya Kasnacheev <ilya.kasnacheev@gmail.com<mailto:ilya.kasnacheev@gmail.com>>
Отправлено: 30 августа 2019 г. 15:49
Кому: user@ignite.apache.org<mailto:user@ignite.apache.org> <user@ignite.apache.org<mailto:user@ignite.apache.org>>
Тема: Re: Perfomance drop

Hello!

Have you tried collecting thread dumps/JFR/etc while such an operation is waiting? It could
provide some clues.

Do you have GC log collected? Any long GC pauses?

Regards,
--
Ilya Kasnacheev


пт, 30 авг. 2019 г. в 14:43, Кузин Никита (Nikita Kuzin) <nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>>:
Hi!

We have application that goes to ignite node (only one node in grid).
Application and ignite are located on one machine.

And we noticed that sometimes that cache located at ignite is very slow: we execute method
igniteCache.containsKey(key) and it can lasts 900ms, size of cache less than 10000 entries
This cache is used mostly for read.

Persistence is off
All of JVM recommendations is applied
OS: windows server 2012 R2

_________________________________
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nkuzin@iitdgroup.ru<mailto:nkuzin@iitdgroup.ru>
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

Mime
View raw message