ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Scherbakov (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (IGNITE-3195) Rebalancing: IgniteConfiguration.rebalanceThreadPoolSize is wrongly treated
Date Tue, 27 Aug 2019 16:41:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916855#comment-16916855
] 

Alexei Scherbakov edited comment on IGNITE-3195 at 8/27/19 4:40 PM:
--------------------------------------------------------------------

[~avinogradov]

Overall fix looks good, but I think we could improve it.

1. Looks like it's safe to remove ordering for historical rebalance because after IGNITE-10078
rmvQeue for partition is no longer cleared during rebalance and removals cannot be lost.
Given what, we could use single thread pool for historical and full rebalance and parallelize
historical rebalance on supplier side same as full.
This is right thing to do because from user side of view there is no difference between full
and historical rebalance and they can happen simultaneously.
Note, proper fix for writing tombstones is on the way [1]

2. Looks like current implementation for detecting partition completion on concurrent processing
using *queued*and *processed* is flawed.
Consider the scenario:
Demander sends initial demand request for single partition.
Supplier replies with 2 total supply messages which are starting to process in parallel.
2-d message is last.
2-d message started to process first, increments *queued* to N (number of entries in message)
2-d message finished processing, incrementing *processed* to N.
Because this is last message partition will be owned before other messages are applied.

[1] https://issues.apache.org/jira/browse/IGNITE-11704


was (Author: ascherbakov):
[~avinogradov]

Overall fix looks good, but I think we could improve it.

1. Looks like it's safe to remove ordering for historical rebalance because after IGNITE-10078
rmvQeue for partition is no longer cleared during rebalance and removals cannot be lost.
Given what, we could use single thread pool for historical and full rebalance and parallelize
historical rebalance on supplier side same as full.
This is right thing to do because from user side of view there is no difference between full
and historical rebalance and they can happen simultaneously.
Note, proper fix for writing tombstones is on the way [1]

2. Looks like current implementation for detecting partition completion on concurrent processing
using *queued *and *processed *is flawed.
Consider the scenario:
Demander sends initial demand request for single partition.
Supplier replies with 2 total supply messages which are starting to process in parallel.
2-d message is last.
2-d message started to process first, increments *queued *to N (number of entries in message)
2-d message finished processing, incrementing *processed *to N.
Because this is last message partition will be owned before other messages are applied.

[1] https://issues.apache.org/jira/browse/IGNITE-11704

> Rebalancing: IgniteConfiguration.rebalanceThreadPoolSize is wrongly treated
> ---------------------------------------------------------------------------
>
>                 Key: IGNITE-3195
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3195
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache
>            Reporter: Denis Magda
>            Assignee: Anton Vinogradov
>            Priority: Major
>              Labels: iep-16
>             Fix For: 2.8
>
>          Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Presently it's considered that the maximum number of threads that has to process all
demand and supply messages coming from all the nodes must not be bigger than {{IgniteConfiguration.rebalanceThreadPoolSize}}.
> Current implementation relies on ordered messages functionality creating a number of
topics equal to {{IgniteConfiguration.rebalanceThreadPoolSize}}.
> However, the implementation doesn't take into account that ordered messages, that correspond
to a particular topic, are processed in parallel for different nodes. Refer to the implementation
of {{GridIoManager.processOrderedMessage}} to see that for every topic there will be a unique
{{GridCommunicationMessageSet}} for every node.
> Also to prove that this is true you can refer to this execution stack 
> {noformat}
> java.lang.RuntimeException: HAPPENED DEMAND
> 	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:378)
> 	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:364)
> 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:622)
> 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:320)
> 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$300(GridCacheIoManager.java:81)
> 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1125)
> 	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> 	at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:105)
> 	at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2456)
> 	at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1179)
> 	at org.apache.ignite.internal.managers.communication.GridIoManager.access$1900(GridIoManager.java:105)
> 	at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:1148)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> All this means that in fact the number of threads that will be busy with replication
activity will be equal to {{IgniteConfiguration.rebalanceThreadPoolSize}} x number_of_nodes_participated_in_rebalancing



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message