gearpump-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junguk Cho <>
Subject Re: Question about Dynamic Computation DAG
Date Wed, 25 Jan 2017 23:55:08 GMT
Hi, Huafeng and Xu, Qian.

Thank you for reply and sorry for using incorrect e-mail address.

I have following questions.
  2. When dynamic DAG happens, at least once is not guaranteed.
=> Can I ask why it does not guarantee it? Is it because of switching
from  between old task to new task? Does  gearpump

  3. Don't support adding a node or deleting a node.
=> Which means we cannot increase the number of nodes for the specific
tasks? For example, I have split -> sum topology and I want to add two more
nodes which run sum processor.
Is it possible? I am not clear between changing the parallelism and  Don't
support adding a node or deleting a node.

Thanks in advance.
- Junguk

On Mon, Jan 16, 2017 at 12:46 AM, Vincent Wang <> wrote:

> Hi Junguk,
>   Sorry the missing of documentation. Here is a simple example:
>   First, you can submit the *wordcount *application to Gearpump, the
> wordcount jar file is in example folder.If everything works fine, you can
> see a DAG from Gearpump's dashboard
> <>
>   After that, you can right click the node and a menu will pop up. The
> dynamic DAG now support two operations, *changing the parallelism* and *replacing
> the entire processor(node)*.
>   For replacing the processor, user need to supply an other jar file that
> contains the new streaming task's java class.
>   Basically every node in a streaming DAG in Gearpump is called a
> Processor and it's internal business logic should be implemented by a
> corresponding Task by user.
> Internally, when replacing a nodeA with nodeB, Gearpump just create a new
> nodeB in the original DAG at the right place. And then lead the data stream
> to the new nodeB.
>   Here are some limitations of dynamic DAG:
>   1. Don't support changing DataSource or DataSink
>   2. When dynamic DAG happens, at least once is not guaranteed.
>   3. Don't support adding a node or deleting a node.
> Thanks,
> Huafeng
> Xu, Qian <>于2017年1月15日周日 下午1:04写道:
>> Hi Junguk,
>> Thanks for sending me the mail. I'd like you to post your question on the
>> user mailing list (
>> gearpump-user/), so that more people might get involved.
>> Stanley
>> On Sat, Jan 14, 2017 at 3:25 PM, Junguk Cho <> wrote:
>> Hi.
>> I am new to gearpump.
>> While I am reading the document (http://gearpump.incubator.
>>, I wonder
>> Dynamic Computation DAG in gearpump. However, the explanation is not
>> sufficient.
>> Would you explain how it work in details like functions, limitation and
>> interface for reconfiguration?
>> Thanks,
>> Junguk
>> --
>> Qian Xu (Stanley)
>> _______________________________________________________________________
>> This e-mail may contain confidential material for the sole use of the
>> intended recipient(s). Any review or distribution by others is strictly
>> prohibited. If you are not the intended recipient, please contact the
>> sender and delete all copies.

View raw message