gearpump-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Wang <fvunic...@gmail.com>
Subject Re: Question about Dynamic Computation DAG
Date Mon, 16 Jan 2017 07:46:52 GMT
Hi Junguk,

  Sorry the missing of documentation. Here is a simple example:

  First, you can submit the *wordcount *application to Gearpump, the
wordcount jar file is in example folder.If everything works fine, you can
see a DAG from Gearpump's dashboard

<https://lh3.googleusercontent.com/-HLw7fi3ps0k/Vi7gKcnId-I/AAAAAAAAAEc/ujYyrefykCU/s1600/1.PNG>

  After that, you can right click the node and a menu will pop up. The
dynamic DAG now support two operations, *changing the parallelism* and
*replacing
the entire processor(node)*.
  For replacing the processor, user need to supply an other jar file that
contains the new streaming task's java class.

  Basically every node in a streaming DAG in Gearpump is called a Processor
and it's internal business logic should be implemented by a corresponding
Task by user.
Internally, when replacing a nodeA with nodeB, Gearpump just create a new
nodeB in the original DAG at the right place. And then lead the data stream
to the new nodeB.

  Here are some limitations of dynamic DAG:
  1. Don't support changing DataSource or DataSink
  2. When dynamic DAG happens, at least once is not guaranteed.
  3. Don't support adding a node or deleting a node.

Thanks,
Huafeng



Xu, Qian <sx.away@googlemail.com>于2017年1月15日周日 下午1:04写道:

> Hi Junguk,
>
> Thanks for sending me the mail. I'd like you to post your question on the
> user mailing list (
> http://mail-archives.apache.org/mod_mbox/incubator-gearpump-user/), so
> that more people might get involved.
>
> Stanley
>
> On Sat, Jan 14, 2017 at 3:25 PM, Junguk Cho <jmanbal@gmail.com> wrote:
>
> Hi.
>
> I am new to gearpump.
>
> While I am reading the document (
> http://gearpump.incubator.apache.org/releases/latest/introduction/features/index.html),
> I wonder Dynamic Computation DAG in gearpump. However, the explanation is
> not sufficient.
> Would you explain how it work in details like functions, limitation and
> interface for reconfiguration?
>
>
> Thanks,
> Junguk
>
>
>
>
> --
> Qian Xu (Stanley)
> _______________________________________________________________________
>
> This e-mail may contain confidential material for the sole use of the
> intended recipient(s). Any review or distribution by others is strictly
> prohibited. If you are not the intended recipient, please contact the
> sender and delete all copies.
>

Mime
View raw message