gearpump-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junguk Cho <jman...@gmail.com>
Subject Re: Question about Dynamic Computation DAG
Date Wed, 25 Jan 2017 23:55:08 GMT
Hi, Huafeng and Xu, Qian.

Thank you for reply and sorry for using incorrect e-mail address.

I have following questions.
  2. When dynamic DAG happens, at least once is not guaranteed.
=> Can I ask why it does not guarantee it? Is it because of switching
synchronization
from  between old task to new task? Does  gearpump

  3. Don't support adding a node or deleting a node.
=> Which means we cannot increase the number of nodes for the specific
tasks? For example, I have split -> sum topology and I want to add two more
nodes which run sum processor.
Is it possible? I am not clear between changing the parallelism and  Don't
support adding a node or deleting a node.

Thanks in advance.
- Junguk

On Mon, Jan 16, 2017 at 12:46 AM, Vincent Wang <fvunicorn@gmail.com> wrote:

> Hi Junguk,
>
>   Sorry the missing of documentation. Here is a simple example:
>
>   First, you can submit the *wordcount *application to Gearpump, the
> wordcount jar file is in example folder.If everything works fine, you can
> see a DAG from Gearpump's dashboard
>
>
> <https://lh3.googleusercontent.com/-HLw7fi3ps0k/Vi7gKcnId-I/AAAAAAAAAEc/ujYyrefykCU/s1600/1.PNG>
>
>   After that, you can right click the node and a menu will pop up. The
> dynamic DAG now support two operations, *changing the parallelism* and *replacing
> the entire processor(node)*.
>   For replacing the processor, user need to supply an other jar file that
> contains the new streaming task's java class.
>
>   Basically every node in a streaming DAG in Gearpump is called a
> Processor and it's internal business logic should be implemented by a
> corresponding Task by user.
> Internally, when replacing a nodeA with nodeB, Gearpump just create a new
> nodeB in the original DAG at the right place. And then lead the data stream
> to the new nodeB.
>
>   Here are some limitations of dynamic DAG:
>   1. Don't support changing DataSource or DataSink
>   2. When dynamic DAG happens, at least once is not guaranteed.
>   3. Don't support adding a node or deleting a node.
>
> Thanks,
> Huafeng
>
>
>
> Xu, Qian <sx.away@googlemail.com>于2017年1月15日周日 下午1:04写道:
>
>> Hi Junguk,
>>
>> Thanks for sending me the mail. I'd like you to post your question on the
>> user mailing list (http://mail-archives.apache.org/mod_mbox/incubator-
>> gearpump-user/), so that more people might get involved.
>>
>> Stanley
>>
>> On Sat, Jan 14, 2017 at 3:25 PM, Junguk Cho <jmanbal@gmail.com> wrote:
>>
>> Hi.
>>
>> I am new to gearpump.
>>
>> While I am reading the document (http://gearpump.incubator.
>> apache.org/releases/latest/introduction/features/index.html), I wonder
>> Dynamic Computation DAG in gearpump. However, the explanation is not
>> sufficient.
>> Would you explain how it work in details like functions, limitation and
>> interface for reconfiguration?
>>
>>
>> Thanks,
>> Junguk
>>
>>
>>
>>
>> --
>> Qian Xu (Stanley)
>> _______________________________________________________________________
>>
>> This e-mail may contain confidential material for the sole use of the
>> intended recipient(s). Any review or distribution by others is strictly
>> prohibited. If you are not the intended recipient, please contact the
>> sender and delete all copies.
>>
>

Mime
View raw message