gearpump-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Wang <fvunic...@gmail.com>
Subject Re: Question about Dynamic Computation DAG
Date Sun, 29 Jan 2017 08:17:52 GMT
Hi Junguk,

Sorry for the late response.
Let me first answer your second question and I'll still use the Split ->
Sum example. Here adding a node or deleting a node means changing the
topology of the DAG, such as adding a KafkaSink after the Sum processor to
write the result to Kafka. While changing the parallelism, it just means
the runtime task number of Split or Sum. So basically node is an abstract
conception and the parallelism is a runtime conception, which is more like
threads.

And for your first question, when dynamic DAG happens, there will be a
period of time that message delivered to the old tasks and we also don't
maintain tasks' states, so at-least-once is not guaranteed.

Thanks,
Huafeng

Junguk Cho <jmanbal@gmail.com>于2017年1月26日周四 上午7:55写道:

> Hi, Huafeng and Xu, Qian.
>
> Thank you for reply and sorry for using incorrect e-mail address.
>
> I have following questions.
>   2. When dynamic DAG happens, at least once is not guaranteed.
> => Can I ask why it does not guarantee it? Is it because of switching synchronization
> from  between old task to new task? Does  gearpump
>
>   3. Don't support adding a node or deleting a node.
> => Which means we cannot increase the number of nodes for the specific
> tasks? For example, I have split -> sum topology and I want to add two more
> nodes which run sum processor.
> Is it possible? I am not clear between changing the parallelism and  Don't
> support adding a node or deleting a node.
>
> Thanks in advance.
> - Junguk
>
> On Mon, Jan 16, 2017 at 12:46 AM, Vincent Wang <fvunicorn@gmail.com>
> wrote:
>
> Hi Junguk,
>
>   Sorry the missing of documentation. Here is a simple example:
>
>   First, you can submit the *wordcount *application to Gearpump, the
> wordcount jar file is in example folder.If everything works fine, you can
> see a DAG from Gearpump's dashboard
>
>
> <https://lh3.googleusercontent.com/-HLw7fi3ps0k/Vi7gKcnId-I/AAAAAAAAAEc/ujYyrefykCU/s1600/1.PNG>
>
>   After that, you can right click the node and a menu will pop up. The
> dynamic DAG now support two operations, *changing the parallelism* and *replacing
> the entire processor(node)*.
>   For replacing the processor, user need to supply an other jar file that
> contains the new streaming task's java class.
>
>   Basically every node in a streaming DAG in Gearpump is called a
> Processor and it's internal business logic should be implemented by a
> corresponding Task by user.
> Internally, when replacing a nodeA with nodeB, Gearpump just create a new
> nodeB in the original DAG at the right place. And then lead the data stream
> to the new nodeB.
>
>   Here are some limitations of dynamic DAG:
>   1. Don't support changing DataSource or DataSink
>   2. When dynamic DAG happens, at least once is not guaranteed.
>   3. Don't support adding a node or deleting a node.
>
> Thanks,
> Huafeng
>
>
>
> Xu, Qian <sx.away@googlemail.com>于2017年1月15日周日 下午1:04写道:
>
> Hi Junguk,
>
> Thanks for sending me the mail. I'd like you to post your question on the
> user mailing list (
> http://mail-archives.apache.org/mod_mbox/incubator-gearpump-user/), so
> that more people might get involved.
>
> Stanley
>
> On Sat, Jan 14, 2017 at 3:25 PM, Junguk Cho <jmanbal@gmail.com> wrote:
>
> Hi.
>
> I am new to gearpump.
>
> While I am reading the document (
> http://gearpump.incubator.apache.org/releases/latest/introduction/features/index.html),
> I wonder Dynamic Computation DAG in gearpump. However, the explanation is
> not sufficient.
> Would you explain how it work in details like functions, limitation and
> interface for reconfiguration?
>
>
> Thanks,
> Junguk
>
>
>
>
> --
> Qian Xu (Stanley)
> _______________________________________________________________________
>
> This e-mail may contain confidential material for the sole use of the
> intended recipient(s). Any review or distribution by others is strictly
> prohibited. If you are not the intended recipient, please contact the
> sender and delete all copies.
>
>
>

Mime
View raw message