ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilya Lantukh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-8020) Rebalancing for persistent caches should transfer file store over network instead of using existing supply/demand protocol
Date Thu, 22 Mar 2018 13:36:00 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ilya Lantukh updated IGNITE-8020:
---------------------------------
    Description: 
Existing rebalancing protocol is suitable for in-memory data storage, but for data persisted
in files it is sub-optimal and requires a lot of unnecessary steps. Efforts to optimize it
led to necessity to completely rework the protocol - instead of sending batches (SupplyMessages)
with cache entries it is possible to send data files directly.

The algorithm should look like this:
1. Demander node sends requests with required partition IDs (like now)
2. Supplier node receives request and performs a checkpoint.
3. After checkpoint is done, supplier sends files with demanded partitions using low-level
NIO API.
4. During steps 2-3, demander node should work in special mode - it should temporary store
all incoming updates in such way that they can be quickly applied later.
5. After files are transferred, demander applies updates stored at step 4.

The tricky part here is to switch work modes of demander node avoiding all possible race conditions.
Also, the aforementioned algorithm should be extended to transfer or rebuild query indexes.

> Rebalancing for persistent caches should transfer file store over network instead of
using existing supply/demand protocol
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-8020
>                 URL: https://issues.apache.org/jira/browse/IGNITE-8020
>             Project: Ignite
>          Issue Type: Improvement
>          Components: persistence
>            Reporter: Ilya Lantukh
>            Assignee: Ilya Lantukh
>            Priority: Major
>
> Existing rebalancing protocol is suitable for in-memory data storage, but for data persisted
in files it is sub-optimal and requires a lot of unnecessary steps. Efforts to optimize it
led to necessity to completely rework the protocol - instead of sending batches (SupplyMessages)
with cache entries it is possible to send data files directly.
> The algorithm should look like this:
> 1. Demander node sends requests with required partition IDs (like now)
> 2. Supplier node receives request and performs a checkpoint.
> 3. After checkpoint is done, supplier sends files with demanded partitions using low-level
NIO API.
> 4. During steps 2-3, demander node should work in special mode - it should temporary
store all incoming updates in such way that they can be quickly applied later.
> 5. After files are transferred, demander applies updates stored at step 4.
> The tricky part here is to switch work modes of demander node avoiding all possible race
conditions. Also, the aforementioned algorithm should be extended to transfer or rebuild query
indexes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message