Mailing-List: contact user-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@ignite.apache.org
From: =?utf-8?Q?J=C3=B6rn_Franke?= <jornfranke@gmail.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Date: Sat, 15 Oct 2016 13:32:10 +0200
Subject: Re: Market data binary messages processed with Ignite and Spark
Message-Id: <E88B4E19-10FB-4A4F-A5BA-72E552E7A088@gmail.com>
References: <1476509004979-8313.post@n6.nabble.com>
In-Reply-To: <1476509004979-8313.post@n6.nabble.com>
To: user@ignite.apache.org
archived-at: Sat, 15 Oct 2016 11:32:20 -0000

Depends how you do the lookup. Is it by ID? Then keep the ids as small as po=
ssible. Lookup is fastest in a hash map type of datastructure. In case of a d=
istributed setting supported by a bloom filter.
Apache Ignite can be seen as suitable.

Depending on what you need to do (maybe your approach requires hyperlolog st=
ructured etc) you may look also at redis, but from what you describe Ignite i=
s suitable.

I do not think the caching of the filesystem benefits here, because the key i=
s the datastructure here (hash map).
The concrete physical infrastructure to meet your SLAs can only be determine=
d when you experiment with real data.

Maybe you can tell a little bit more about the data. Are the messages depend=
ent? What type of calculation do you do?

> On 15 Oct 2016, at 07:23, Pranas Baliuka <pranas@orangecap.net> wrote:
>=20
> Dear Ignite enthusiasts,
>=20
> I am beginner in Apache Ingnite, but want to prototype solution for using
> Ignite cashes with market data distributed across multiple nodes running
> Spark RDD.
>=20
> I'd like to be able to send sequenced (from 1) binary messages (size from 4=
0
> bytes to max 1 Kb) to custom Spark job processing multidimensional cube of=

> parameters.=20
> Each market data event must be processed once from #1 to #records for each=

> parameter.=20
> Number of messages ~40-50 M in one batch.
>=20
> It would be great if you can share your experience with similar imp.=20
>=20
> My high level thinking:
> * Prepare system by loading Ignite Cashe (unzipping market data drop-copy
> file, converting to preferred binary format and publish IgniteCache<Long,
> BinaryObject>;
> * Spawn Spark job to process input cube of parameters (SparkRDD) each usin=
g
> cashed the same IgniteCashe (accessed sequentially by sequence number from=
 1
> - #messages as key);
> * Store results in RDMS/NoSQL storage;
> * Perform reports from Apache Zeppelin using Spark.R interpreter.
>=20
> I need for Cache outlive Spark jobs i.e. may run different cube of
> parameters after one is finished.
>=20
> I am not sure if Ignite would be able to lookup messages efficiently (I'd
> need ~400 Km/s sustained retrieval).=20
> Or should I consider something more file oriented e.g. use memory mounted
> file system on each node ...
>=20
> Thank in advance to share your ideas/proposals/know-how!
>=20
>=20
>=20
>=20
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.c=
om/Market-data-binary-messages-processed-with-Ignite-and-Spark-tp8313.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.