ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zbyszek <zbab...@yahoo.com>
Subject Private cache?
Date Mon, 12 Mar 2018 14:55:00 GMT
Hello Igniters,

I wonder if anybody could advise on the following.
I am trying to implement simple extract, transform, load (ETL) process using
Apache Ignite 2.3 platform.
In extract phase local caches are loaded with some data and then passed to
transform phase where the SQL SELECT (sometimes quite complex)  is executed
against these local caches (after SQL execution the local caches are
destroyed). Then the result of SQL is uploaded to global partitioned
cluster-wide cache.

The problem is that local caches seem to be rather heavy tool for this
purpose (we are about to process thousands of local caches per day). They
seem not to be 'very local', i.e. involve significant part of cluster
infrastructure (exchange worker, grid timeout workers, etc...), and causing
memory leaks (like IGNITE-7319, IGNITE-7918).

I would like to ask if there is a way to execute SQL queries against some
really lightweight cache layer (i.e. somehow replace the underlying
infrastructure with simple map implementation or something similar).  Also,
would the request for real private cache (not visible to other nodes, not
involving cluster infrastructure, allowing SQL execution) make sense?

thank you in advance,


Sent from: http://apache-ignite-users.70518.x6.nabble.com/

View raw message