ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Vinogradov <avinogra...@gridgain.com>
Subject Re: Rebalancing speedup (IGNITE-1093)
Date Wed, 05 Aug 2015 09:57:09 GMT
Parallel Demand pool seems to be ready.
Working on parallel Supply pool.

General idea is to use systemPool to handle demand requests. Amount ot
parallel workers will be limited indirectly by Demand poll's parallelism
limits.
But overload still can happens in this case. Will write load tests.

Next idea is to go through whole partition and when
s.messageSize() >= cctx.config().getRebalanceBatchSize()
send supply batch, store current iterator, swapLsnr and etc. and return
from supply thread.
When same node will make next request then supplyPool just continue
iterating and give next or last batch for requested partition.

Also speed can be increased if SupplyPool will prepare N batches on first
demand request and will generate additional on each new request.

On Thu, Jul 30, 2015 at 9:41 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> Anton,
>
> Very good point. Valentin is already doing per-partition topic in his work
> for Continuous Queries:
> https://issues.apache.org/jira/browse/IGNITE-104
>
> Perhaps it makes sense for both of you to sync up.
>
> Also, as a side note, whenever starting a dev list discussion about a
> ticket, it always helps to include a link to the dev list thread into Jira
> (I already did it for IGNITE-1093).
>
> D.
>
> On Thu, Jul 30, 2015 at 11:31 AM, Anton Vinogradov <
> avinogradov@gridgain.com
> > wrote:
>
> > Hello,
> >   I think that speed of rebalancing can be increased by parallel handling
> > of supply messages.
> >   Main Idea is to use per-partition topics.
> >
> >   At the start of DemandPool we can create topics for every partition and
> > handle each partition's supply messages at separated listener. In this
> case
> > only one thread will request lock for partition it have to relocate.
> >
> >   Also we can do the same at SupplyPool, and make supply messages in
> > parallel way.
> >
> >   Amount of simultaneous messages should be limited to prevent
> overloading.
> > Current idea is to request N partitions at start and request one next
> > partition when one of prevous relocated.
> >
> >   Per-partition listeners can be created this way:
> > for (int p = 0; p <= maxP; p++) {
> >     cctx.io().addOrderedHandler(topic(p), new CI2<UUID,
> > GridDhtPartitionSupplyMessage>() {
> >         @Override public void apply(UUID nodeId,
> > GridDhtPartitionSupplyMessage msg) {
> >             handleSupplyMessage(new SupplyMessage(nodeId, msg));
> >         }
> >     });
> > }
> >
> > Thoughts?
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message