ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Ignite Load Performance when memory exhausted
Date Thu, 25 Jun 2020 10:04:01 GMT
Hello!

Until you got this message, you did not need to catch up with disk (except
for WAL writing, maybe), but once you see it, good days are over and you
need to do checkpoints with all pages which were updated recently.

There are things to tune here, but don't expect miracles.

https://apacheignite.readme.io/docs/durable-memory-tuning

Regards,
-- 
Ilya Kasnacheev


вт, 23 июн. 2020 г. в 18:25, njcstreet <njcstreet@gmail.com>:

> Hi,
>
> I am evaluating Apache Ignite for a use case where I know we may not have
> enough RAM to store the entire database. For example we might store 30 days
> of data, but only have enough memory for 15, and in fact most of the
> queries
> will be against the last 5 days of data. I am doing some tests loading the
> 30 days of data (roughly 2 billion objects) into a test environment.
>
>
>
> ·         The test environment has 6 servers each with 128GB RAM, which I
> am
> allocating roughly 70% to Ignite to allow the OS enough memory.
>
> ·         This is using Native Persistence in Ignite 2.8.1.
>
> ·         The disks I am using are not as fast as production would be (they
> are not SSD) but they are still reasonably fast.
>
> ·         The object I am storing is reasonably simple, it has 25
> properties
> which are integers (2 of these have indexes on) and 4 which are floats, but
> there are a lot of them (roughly 2 billion).
>
> ·         The cache the data is being loaded into is partitioned.
>
> ·         Data is loaded using IDataStreamer – there is a single loader
> process which has 10 threads reading files, then submitting into a single
> DataStreamer.
>
>
>
> What I am observing is that:
>
>
>
> ·         Whilst the default region has RAM available, the inserts are
> quite
> quick – even though the data is being written to disk (the WAL is
> enabled).
> I get on average 150K objects inserted per second.
>
> ·         Once the default region runs out of RAM, I see an error message
>
> “WARN|org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl|Page
> replacements started, pages will be rotated with disk, this will affect
> storage performance (consider increasing
> DataRegionConfiguration#setMaxSize).”
>
> ·         After this, inserts are much slower, going down to about 19K
> objects per second.
>
>
>
> Now I kind of understand what is happening here is that for every new page
> in memory that needs to be allocated, Ignite is having to purge another
> page
> from memory (this is implied by the error message). But what I don’t
> understand is why the performance impact is so great (150K objects / sec
> when memory is available vs 19K per sec when memory is exhausted). Before
> and after the region ran out of RAM, the data was being written to disk.
> After the region ran out of RAM, I would expect the only difference to be
> that it needs to purge some data from RAM to make space for the data that
> is
> loading – I wouldn’t have thought there should be any more disk activity
> since through Ignite Persistence the data was already being written to
> disk.
> Maybe I have something mis-configured, or I have misunderstood something?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Mime
View raw message