ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mammo, Mulugeta" <mulugeta.ma...@intel.com>
Subject Adding Persistent Memory Support for Ignite
Date Thu, 16 Nov 2017 15:16:47 GMT
Hi all,



Ignite, when persistence mode is enabled, stores data and indexes on disk. To minimize the
latency of disks, several tuning options can be applied. Setting the page size of a memory
region to match the page size of the underlying storage, using a separate disk for the WAL,
and using production-level SSDs are just a few of them [ https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning
].

A persistent memory store with low latency and high capacity offers a viable alternative to
disks. In light of this, we are proposing to make use of our Low Level Persistent Library
(LLPL), https://github.com/pmem/pcj/tree/master/LLPL, to offer a persistent memory storage
for Ignite.

At this point, we envision two distinct implementation options:

1.      Data and indexes will continue to be stored in the off-heap memory but the disk will
be replaced by a persistent memory. Since persistence memory in this option is not a file
system, the logic currently offered by WAL file and the partition files would have to be implemented
from scratch.

2.      In this option, we eliminate the current check-point process and the WAL file. We
will use a memory region defined by LLPL to store data and indexes. There will be no off-heap
memory. DRAM will be exclusively used to store hot cache entries just like the on-heap cache
is in the current implementation.

In both cases, there are more details and subtleties that have to handled - e.g. the atomic
and transactional guarantees offered. More clarifications will be given as we go along. And,
feel free to provide your thoughts.

Thanks,
Mulugeta


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message