ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From msuh <m...@jobcase.com>
Subject QueryCursor checkpoint
Date Thu, 24 Jan 2019 19:03:35 GMT

Our end production cluster would be working with many billions of entities
in many caches, and have use cases where we need to run ScanQuery over an
entire cache to update certain fields. 

We expect that there could definitely be failures in the middle of a single
ScanQuery due to the sheer size of the caches. Since we wouldn't want to
rerun ScanQuery from the start, we're wondering if we could keep some
checkpoint of up to which point we've processed in the QueryCursor. The
QueryCursor API doesn't seem to show any methods that allow that, but I may
not be looking at the right place? Would there be any other efficient ways
to keep track of vaguely up to which point we've processed? If QueryCursor
doesn't provide anything externally, would partition number be the best

But from what I've seen, it seemed like entities in partitions shift around
(from rebalancing or something?), so not sure if that's even possible.

Sent from: http://apache-ignite-users.70518.x6.nabble.com/

View raw message