gearpump-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <>
Subject Re: Regarding dynamic DAG
Date Tue, 13 Sep 2016 02:53:39 GMT

Internally, each Task has a lifetime and subscription list (successor
tasks) both of which can be changed.
For your example, all "B" tasks will update their lifetimes and stop
sending messages after the end of their lifetimes. New "X" tasks will be
launched and send messages.  For predecessor tasks, their subscription list
will be updated to "X" and they start to send messages to "X" tasks.
Automatically, successor tasks will receive messages from "X" tasks.

You can change the parallelism at runtime but not the partitioning.

Manu Zhang

On Tue, Sep 13, 2016 at 5:25 AM Hyunseok Chang <>

> Hi,
> I'd like to know more about the dynamic DAG feature.
> Let's say I have a DAG of:  source -> A -> B -> C.  I want to replace "B"
> with "X" node in this chain.
> How does node replacement happen internally?
> Each processor consists of multiple parallel tasks, so node replacement
> should involve killing multiple concurrent tasks for B, and somehow
> introducing new tasks for X without affecting predecessor/successor tasks.
> I'd like to know how this is done internally.
> Also, can I change the parallelism (# of tasks) or type of partitioning
> (hash <-> shuffle) of each processor dynamically at run time?
> Thanks,
> -hs

View raw message