ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mfronczyk <fronc...@gmail.com>
Subject Re: Can Ignite be configured to provide an array like replicated and durable map?
Date Thu, 10 Sep 2015 11:02:46 GMT
Do I need to use EntryProcessor and invoke if I'll guarantee that all data
is directly accessible by the client application? My idea is to run Ignite
embedded in the client, instead of as a remote cluster, to minimise the
latency, and to have a single backup node to which the data will be
replicated. Invoke seems like an API to execute operations on the cluster.

Storing 1 billion elements in separate keys isn't probably the way to go.
As you said, it'll take much more memory and fetching 16k of objects will
be slow. If I store 16384 array elements as one cache entry, how will it
work when a single element is modified? Will it have to replicate the whole
array to the backup node and will it cause writing of the whole array to
the permanent data store?

On Thu, Sep 10, 2015 at 4:23 AM, dsetrakyan [via Apache Ignite Users] <
ml-node+s70518n1329h40@n6.nabble.com> wrote:

> mfronczyk wrote
> I need a data structure that holds 1 billion of objects. Each object will
> consist of 4 integers. This will be used to model a large 2-dimensional
> grid.
> To keep a data structure this big in memory, you should consider using
> Ignite Off-Heap Memory support [1]. Also make sure that you have enough
> memory on your server to fit the whole data set.
> [1] http://apacheignite.gridgain.org/docs/off-heap-memory
> mfronczyk wrote
> Another important feature is that the memory layout of the structure
> matches the array indexes - element with index n is before the element n +
> 1. There will be a need to read 16384 consecutive elements quickly and that
> memory layout will be very cache friendly for such operation.
> If you remove this consecutive memory layout requirement, then you can
> actually store 1 billion of objects separately, using array index as a key.
> In this case, you can use full IgniteCache API, access each element
> separately, and update different elements concurrently. Depending on your
> requirements, this approach may be easier to use. However, this approach
> will also consume more memory than the approach suggested by Valentin.
> mfronczyk wrote
> Elements written to the structure need to be durable. The plan is to
> replicate the writes on 2 nodes. Writes to about 1% of the indexes are more
> important than others and it should be guaranteed that acknowledged writes
> for that indexes survive a complete crash of one node. There should be no
> time window during which such writes are lost. But it's OK for the rest of
> 99% other writes to have a chance of data loss for performance reasons. So
> we need async replication with a possibility to enable synchronous
> replication for certain puts.
> Synchronous vs. asynchronous backups can be configured on per-cache level
> via FULL_SYNC and PRIMARY_SYNC modes [2]. So if you really need to separate
> synchronous updates from asynchronous updates, you will have to split your
> data structure into 2 parts and store it in 2 different caches. My
> preference would be that you start off with one cache in FULL_SYNC mode,
> and introduce another one in PRIMARY_SYNC mode only if you run into
> performance issue.
> [2]
> http://apacheignite.gridgain.org/v1.3/docs/primary-and-backup-copies#synchronous-and-asynchronous-backups
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1329.html
> To unsubscribe from Can Ignite be configured to provide an array like
> replicated and durable map?, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1304&code=ZnJvbmN6eWtAZ21haWwuY29tfDEzMDR8LTIwNjU4OTMyNTc=>
> .
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

MichaƂ Fronczyk

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1338.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
View raw message