ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dsetrakyan <dsetrak...@apache.org>
Subject Re: Can Ignite be configured to provide an array like replicated and durable map?
Date Thu, 10 Sep 2015 02:23:10 GMT
mfronczyk wrote
> I need a data structure that holds 1 billion of objects. Each object will
> consist of 4 integers. This will be used to model a large 2-dimensional
> grid.

To keep a data structure this big in memory, you should consider using
Ignite Off-Heap Memory support [1]. Also make sure that you have enough
memory on your server to fit the whole data set.

[1] http://apacheignite.gridgain.org/docs/off-heap-memory


mfronczyk wrote
> Another important feature is that the memory layout of the structure
> matches the array indexes - element with index n is before the element n +
> 1. There will be a need to read 16384 consecutive elements quickly and
> that memory layout will be very cache friendly for such operation.

If you remove this consecutive memory layout requirement, then you can
actually store 1 billion of objects separately, using array index as a key.
In this case, you can use full IgniteCache API, access each element
separately, and update different elements concurrently. Depending on your
requirements, this approach may be easier to use. However, this approach
will also consume more memory than the approach suggested by Valentin.


mfronczyk wrote
> Elements written to the structure need to be durable. The plan is to
> replicate the writes on 2 nodes. Writes to about 1% of the indexes are
> more important than others and it should be guaranteed that acknowledged
> writes for that indexes survive a complete crash of one node. There should
> be no time window during which such writes are lost. But it's OK for the
> rest of 99% other writes to have a chance of data loss for performance
> reasons. So we need async replication with a possibility to enable
> synchronous replication for certain puts.

Synchronous vs. asynchronous backups can be configured on per-cache level
via FULL_SYNC and PRIMARY_SYNC modes [2]. So if you really need to separate
synchronous updates from asynchronous updates, you will have to split your
data structure into 2 parts and store it in 2 different caches. My
preference would be that you start off with one cache in FULL_SYNC mode, and
introduce another one in PRIMARY_SYNC mode only if you run into performance
issue.

[2]
http://apacheignite.gridgain.org/v1.3/docs/primary-and-backup-copies#synchronous-and-asynchronous-backups



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1329.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message