ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roger Fischer (CW)" <rfis...@Brocade.com>
Subject RE: Strange problems with Ignite native Persistence when Data exceeds Memory
Date Tue, 15 Aug 2017 23:22:02 GMT
Hi Dmitry,

yes, the data set is bigger than the available memory. I am evaluating native persistence,
specifically SQL queries when pages have to be loaded from disk. Thus I need a data set that
is significantly bigger than what fits the memory.

I made another attempt, loading smaller batches and leaving some time between batches for
things to settle down. All went well.

Thus the problem seems to be related to concurrency and/or write workload. The fact that the
3 VMs all use the same physical disk (in the host) does not help the situation.

No, there are no duplicate keys. The exact same data set loads fine when slowing down the
load.

As I mentioned, there were no logs indicating any connection, I/O or other issues.

Unfortunately I have to move on, and will not be able to rerun the scenario with additional
logging at this time. Should I get to it later, I will update this thread.

Thanks...

Roger

-----Original Message-----
From: dkarachentsev [mailto:dkarachentsev@gridgain.com] 
Sent: Tuesday, August 15, 2017 12:04 AM
To: user@ignite.apache.org
Subject: Re: Strange problems with Ignite native Persistence when Data exceeds Memory

Hi Roger,

It looks like not all data could fit in memory and pages started to swap into disk. This means
that too many pages marked as dirty that causes more often and longer checkpointing, that
with WAL writes may significantly slowdown data uploading. Please attach Ignite and GC logs
and try to set
PersistenceStoreConfiguration.setWalMode() to LOG_ONLY or BACKGROUND (this will reduce disk
loading).

Data loss looks strange, is it possible that you have duplicated entry keys?
DataStreamer by default does not override existed entries for performance reasons.

Thanks!
-Dmitry.



--
View this message in context: https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_Strange-2Dproblems-2Dwith-2DIgnite-2Dnative-2DPersistence-2Dwhen-2DData-2Dexceeds-2DMemory-2Dtp16187p16194.html&d=DwICAg&c=IL_XqQWOjubgfqINi2jTzg&r=1esZO0r0bYS90lcsaLA6N4AFxuNo6lzauhETGwdJQoQ&m=VSN0OFdZMcwOzWGy-IVLaRlPAH6MtsPxxyMYlqibzrE&s=IROToNMs5CsGrCism3IEjw-DOnrc3y9N9O7tcBzXnUU&e=
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message