flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: Send events to parallel operator instances
Date Wed, 03 Jun 2015 11:32:28 GMT
Yes, we define a delta function from the first element to the last element
in a window. Now let's discretize the stream using this semantics in
parallel.

Aljoscha Krettek <aljoscha@apache.org> ezt írta (időpont: 2015. jún. 3.,
Sze, 12:20):

> Ah ok. And by distributed you mean that the element that starts the
> window can be processed on a different machine than the element that
> finishes the window?
>
> On Wed, Jun 3, 2015 at 12:11 PM, Gyula Fóra <gyula.fora@gmail.com> wrote:
> > This is not connected to the current implementation. So lets not talk
> about
> > that.
> >
> > This is about theoretical limits to support distributed delta policies
> > which has far reaching implications for the windowing policies one can
> > implement in a prallel way.
> >
> > But you are welcome to throw in any constructive ideas :)
> > On Wed, Jun 3, 2015 at 11:49 AM Aljoscha Krettek <aljoscha@apache.org>
> > wrote:
> >
> >> Part of the reason for my question is this:
> >> https://issues.apache.org/jira/browse/FLINK-1967. Especially my latest
> >> comment there. If we want this, I think we have to overhaul the
> >> windowing system anyways and then it doesn't make sense to explore
> >> complicated workarounds for the current system.
> >>
> >> On Wed, Jun 3, 2015 at 11:07 AM, Gyula Fóra <gyula.fora@gmail.com>
> wrote:
> >> > There are simple ways of implementing it in a non-distributed or
> >> > inconsistent fashion.
> >> > On Wed, Jun 3, 2015 at 8:55 AM Aljoscha Krettek <aljoscha@apache.org>
> >> wrote:
> >> >
> >> >> This already sounds awfully complicated. Is there no other way to
> >> >> implement the delta windows?
> >> >>
> >> >> On Wed, Jun 3, 2015 at 7:52 AM, Gyula Fóra <gyula.fora@gmail.com>
> >> wrote:
> >> >> > Hi Ufuk,
> >> >> >
> >> >> > In the concrete use case I have in mind I only want to send events
> to
> >> >> > another subtask of the same task vertex.
> >> >> >
> >> >> > Specifically: if we want to do distributed delta based windows
we
> >> need to
> >> >> > send after every trigger the element that has triggered the current
> >> >> window.
> >> >> > So practically I want to broadcast some event regularly to all
> >> subtasks
> >> >> of
> >> >> > the same operator.
> >> >> >
> >> >> > In this case the operators would wait until they receive this
event
> >> so we
> >> >> > need to make sure that this event sending is not blocked by the
> actual
> >> >> > records.
> >> >> >
> >> >> > Gyula
> >> >> >
> >> >> > On Tuesday, June 2, 2015, Ufuk Celebi <uce@apache.org> wrote:
> >> >> >
> >> >> >>
> >> >> >> On 02 Jun 2015, at 22:45, Gyula Fóra <gyfora@apache.org
> >> <javascript:;>>
> >> >> >> wrote:
> >> >> >> > I am wondering, what is the suggested way to send some
events
> >> >> directly to
> >> >> >> > another parallel instance in a flink job? For example
from one
> >> mapper
> >> >> to
> >> >> >> > another mapper (of the same operator).
> >> >> >> >
> >> >> >> > Do we have any internal support for this? The first thing
that
> we
> >> >> thought
> >> >> >> > of is iterations but that is clearly an overkill.
> >> >> >>
> >> >> >> There is no support for this at the moment. Any parallel instance?
> >> Or a
> >> >> >> subtask instance of the same task?
> >> >> >>
> >> >> >> Can you provide more input on the use case? It is certainly
> possible
> >> to
> >> >> >> add support for this.
> >> >> >>
> >> >> >> If the events don't need to be inline with the records, we
can
> easily
> >> >> >> setup the TaskEventDispatcher as a separate actor (or extend
the
> task
> >> >> >> manager) to process both backwards flowing events and in general
> any
> >> >> events
> >> >> >> that don't need to be inline with the records. The task deployment
> >> >> >> descriptors need to be extended with the extra parallel instance
> >> >> >> information.
> >> >> >>
> >> >> >> – Ufuk
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message