Mailing-List: contact user-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@ignite.apache.org
Date: Mon, 19 Oct 2015 04:40:55 -0700 (PDT)
From: edwardk <ekiplingg@gmail.com>
To: user@ignite.apache.org
Message-ID: <1445254855436-1647.post@n6.nabble.com>
Subject: Bulk load data at startup
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

I have a requirement to bulk load a lot of Master data which would be common
to entire system.

Since the Data store returns a huge number of rows, I feel the system might
hang if we try to load all of them to memory.

I am assuming it is possible , but not sure , if is is possible to load the
data to a cluster of ignite nodes while fetching from database , in such a
way , we do not utilize all available memory in the node on which the load
is happening and at the same time distribute it across the nodes.

What I mean is say if we have a data of size 500 MB to be loaded, can we
distribute it across 5 nodes of 100 MB each during fetch time from database.

Also, what would be the cache mode for this scenario. I feel replicated
would not be better.But, not sure if partitioned mode is also better here,
as  in partitioned mode only one node has the data and backups of that data
is in other nodes.

So, am not sure what caching mode to use for this.

Can someone help me on that approach to follow using apache ignite for this
scenario.


--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Bulk-load-data-at-startup-tp1647.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.