ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Slow reads in one cache during batch write on different cache in different region
Date Tue, 09 Jul 2019 15:13:40 GMT
Hello!

I think you should collect thread dumps from all nodes to locate the
bottleneck. Then, maybe you need to adjust thread pool sizes.

My idea here is that some thread pool (stripe, probably) gets full with
persistent cache writing operations, and reads have to wait in queue.

Regards,
-- 
Ilya Kasnacheev


вт, 9 июл. 2019 г. в 18:08, Evgeny Pryakhin <breath1988@gmail.com>:

> Hello. I need some help or advise on my problem.
>
> Preface:
> - Ignite 2.5 (I have no options about upgrade to newer version)
> - cluster with 8 servers, 4 CPU and 64GB RAM each, HDD (not SSD).
> Operating system was tuned according to performance guide.
> - two memory regions configured: one in-memory only (500MB) and one with
> persistence enabled (about 40GB memory).
> - one cache in in-memory region (about 300k records), backups - 3. Write
> mode: PRIMARY_SINC.
> - one cache in region with persistence (about 50M records), backups 3.
> Write mode: PRIMARY_SINC.
> - Ignite Thin Client as a driver.
>
> Scenario:
> - I have batch writes on first in memory cache - about 500/sec.
> Continuously.
> - I have a lot of reads on first in-memory cache - about 3k/sec.
> Continuously.
> - I have a lot of batch writes on second persistent cache. Batch size is
> about 1k records. Continuously.
>
> The Problem:
> - when I have batch writes to the second (persistent) cache disabled reads
> from first cache works well with small latency - <1ms.
> - when batch writes to persistent cache is turned on - reads from the
> first cache become very slow - about 200-300ms.
>
> I have no ideas how even to start investigation on this problem. May be I
> can check some metrics of cluster or system metrics on harware servers to
> find the right way to solve my problem. Do you have ant ideas about this?
>

Mime
View raw message