ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Berry <chriswbe...@gmail.com>
Subject Ignite Design questions
Date Tue, 20 Dec 2016 16:48:36 GMT
Greetings,

I am hoping for some design advice? I am new to Ignite and don’t want to start out on the
wrong foot.

At my day job we have a “Rates Engine”. It computes the current Rate for a particular
User request. It is a high volume/low latency system. And currently it lives within a Java
Servlet Engine (jetty), but is moving to Netty.

This is how it works – from 50000 ft :

In the Foreground:

User X requests a Rate (current price) for Product Y.  (Actually it is for Products[X,Y,Z,…n],
where n <= 1000)
The request is routed to “Shard P” based on Y.
The Rate is computed based on the cached RateInfo (essentially, a concurrent HashMap) for
Y, found in Shard P. The computation is relatively expensive CPU-wise.
The Rate is returned to User X. 


In the Background:

The system listens to “Rate change events” for all Products.
Rate changes are read and are aggregated over some period of time. 
Periodically, the appropriate, cached RateInfo is updated as required, in the appropriate
Shard

Simple. And it works well.

Except the system relies on static host bindings. The system is showing its age. 
Moreover, we are moving all of this to AWS.

So, since we have to refactor anyway, we decided to look at better ways. And Ignite seems
a perfect fit.
Ignite clustering, sharding, and replication are all quite appealing.

So my question. How would y’all model this system?

I would like to use Ignite as a standalone cluster (not in-memory within the Rates Engine)
To basically divide the system into its component pieces; A “Rates Engine”, a Distributed
HashMap, and a “Rates Injector”

The things I find most puzzling are:

*  How does a Clustered & Partitioned IgniteCache handle a `cache.getAll(Set<> keys)`
behind the scenes ??
Will it transparently fan those requests out to the appropriate Partition for me (from the
Client – or is it a “double hop” — once to the Cluster, another to the appropriate
shard)??
How does it handle individual failures?? I can’t seem to find much detail about how all
of the internal routing is handled in Ignite.

* There is currently ~30GB of Rates cached. What is an optimal “shard size”? I’ve found
in other sharded systems that; (shard <= 5GB) is a good operational rule of thumb. For
when data has to start moving about for maintenance, etc anything larger than 5GB is cumbersome.
 Is this true for Ignite? I am trying to figure out how big a cluster to build out. 

* Would you build your own “artificial sharding” on top of the Rates. In other words;
apply “affinity collocation” and use say Geo info. Something/anything so that the Rates
collocate by some auxiliary key. That builds reasonably sized shards?? This is probably a
given.

* Is it possible to configure Ignite to, essentially, “always serve data”. E.g. even it
if split-brains?? In other words, since my RatesEngine is read-only, and really doesn’t
care (Rates are assumed to be eventually consistent) – as long as it can find Rates it will
be happy. Only the Rates Injector is really concerned about fidelity. And may blow up. And
as we piece back together the world, the Rates Engine can keep chugging along somehow. Does
that make sense?? Perhaps another way to ask this is, what are the failure scenarios for an
Ignite Cluster??

Sorry if these questions are too broad. I am a total newbie.
I am looking to start a POC, and need a place to begin.

Thanks much,
-- Chris

 

 
Mime
View raw message