ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Wilson <raymond_wil...@trimble.com>
Subject RE: Write behind using Grid Gain
Date Tue, 13 Jun 2017 21:02:33 GMT
Hi Pavel,

It’s a little complicated. The system is essentially a DB in its own right;
actually it’s an IMDG a bit like Ignite, but developed 8 years ago to
fulfill a need we had.

Today, I am looking to modernize that system and rather than continuing to
build and maintain all the core ‘infrastructure’ features of an IMDG such
as clustering, messaging, enterprise caching etc, I am looking to see how
well Ignite fits by running a Proof of Concept project. It turns out it
fits quite well, largely because the architectural structure of both
systems (ie: IMDG) is well aligned in terms of the problems being solved.

The primary gap between the legacy system and IMDG is that IMDG does not
support persistence. The legacy system has a distributed cache that stores
objects that are aggregate collections (10’s of thousand’s) of relatively
simple spatial data records that are operated on by the clustered compute
engine. Sometimes billions of records need to be processed to satisfy a
single query. Your standard run of the mill SQL DB finds these sorts of
queries hard.

I suppose you could use another DB (MS-SQL, AWS:RDS etc) to store those
aggregate blobs, but it seems like a bit of a ‘miss-use case’ when what I’m
really after is a persistence/storage layer J



*From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
*Sent:* Wednesday, June 14, 2017 1:59 AM
*To:* user@ignite.apache.org
*Subject:* Re: Write behind using Grid Gain

Hi Raymond,

I think your use case fits well into traditional Ignite model of
write-through cache store with backing database.

Why do you want to avoid a DB? Do you plan to store data on disk directly
as a set of files?


On Mon, Jun 12, 2017 at 2:14 AM, Raymond Wilson <raymond_wilson@trimble.com>

Hi Pavel,

Thanks for the blog – it explains it quite well.

I have a slightly different use case where groups of records within a much
larger data set are clustered together for efficiency (ie: each of the
cached items in the Ignite grid cache has significant internal structure).
You can think of them as a large number of smallish files (a few Kb to a
few Mb), but file systems don’t like lots of small files.

I have a legacy implementation that houses these small files within a
single larger file, but wanted to know if there was a clean way of
supporting the same structure using the Ignite read/write through support,
perhaps with another system providing relatively transparent persistency
semantics but which does not use a DB to store the data.



*From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
*Sent:* Saturday, May 27, 2017 5:03 AM
*To:* user@ignite.apache.org

*Subject:* Re: Write behind using Grid Gain

I've decided to write a blog post, since this topic seems to be in demand:




Let me know if this helps!

On Fri, May 26, 2017 at 3:50 PM, Chetan D <cchetands@gmail.com> wrote:

Thank you Pavel.

waiting for your response.

On Fri, May 26, 2017 at 6:03 PM, Pavel Tupitsyn <ptupitsyn@apache.org>

To give everyone context, this is not about GridGain, but about Apache

The blog post in question is

Chetan, I'll prepare an example with Ignite 2.0 / ado.net and post it some
time later.


On Fri, May 26, 2017 at 2:32 PM, Chetan D <cchetands@gmail.com> wrote:

++ User List

any help much appreciated.

Thanks And Regards

Chetan D

---------- Forwarded message ----------
From: *Pavel Tupitsyn* <ptupitsyn@apache.org>
Date: Fri, May 26, 2017 at 4:38 PM
Subject: Re: Write behind using Grid Gain
To: Chetan D <cchetands@gmail.com>

Hi Chetan, can you please write this to our user list,

So that entire community can participate.



On Fri, May 26, 2017 at 1:35 PM, Chetan D <cchetands@gmail.com> wrote:

Hi Pavel Tupitsyn,

I have posted a comment in your blog as well (entity framework as ignite
.net store) regarding write behind using ignite.

I have been working on a project where i need to implement distributed
caching and i have been asked to look into grid gain.

This is is the first time i am working on caching and this is entirely new
topic for me.

The example which you have shared i was able to understand a little and the
sad part is even entity framework also i have never worked on.

It would be helpful if you can share me a simple example using ado.net
implementing read through, write through and write behind even a simple
table helps me understand the concept.

View raw message