flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: Send events to parallel operator instances
Date Wed, 03 Jun 2015 10:11:26 GMT
This is not connected to the current implementation. So lets not talk about
that.

This is about theoretical limits to support distributed delta policies
which has far reaching implications for the windowing policies one can
implement in a prallel way.

But you are welcome to throw in any constructive ideas :)
On Wed, Jun 3, 2015 at 11:49 AM Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Part of the reason for my question is this:
> https://issues.apache.org/jira/browse/FLINK-1967. Especially my latest
> comment there. If we want this, I think we have to overhaul the
> windowing system anyways and then it doesn't make sense to explore
> complicated workarounds for the current system.
>
> On Wed, Jun 3, 2015 at 11:07 AM, Gyula Fóra <gyula.fora@gmail.com> wrote:
> > There are simple ways of implementing it in a non-distributed or
> > inconsistent fashion.
> > On Wed, Jun 3, 2015 at 8:55 AM Aljoscha Krettek <aljoscha@apache.org>
> wrote:
> >
> >> This already sounds awfully complicated. Is there no other way to
> >> implement the delta windows?
> >>
> >> On Wed, Jun 3, 2015 at 7:52 AM, Gyula Fóra <gyula.fora@gmail.com>
> wrote:
> >> > Hi Ufuk,
> >> >
> >> > In the concrete use case I have in mind I only want to send events to
> >> > another subtask of the same task vertex.
> >> >
> >> > Specifically: if we want to do distributed delta based windows we
> need to
> >> > send after every trigger the element that has triggered the current
> >> window.
> >> > So practically I want to broadcast some event regularly to all
> subtasks
> >> of
> >> > the same operator.
> >> >
> >> > In this case the operators would wait until they receive this event
> so we
> >> > need to make sure that this event sending is not blocked by the actual
> >> > records.
> >> >
> >> > Gyula
> >> >
> >> > On Tuesday, June 2, 2015, Ufuk Celebi <uce@apache.org> wrote:
> >> >
> >> >>
> >> >> On 02 Jun 2015, at 22:45, Gyula Fóra <gyfora@apache.org
> <javascript:;>>
> >> >> wrote:
> >> >> > I am wondering, what is the suggested way to send some events
> >> directly to
> >> >> > another parallel instance in a flink job? For example from one
> mapper
> >> to
> >> >> > another mapper (of the same operator).
> >> >> >
> >> >> > Do we have any internal support for this? The first thing that
we
> >> thought
> >> >> > of is iterations but that is clearly an overkill.
> >> >>
> >> >> There is no support for this at the moment. Any parallel instance?
> Or a
> >> >> subtask instance of the same task?
> >> >>
> >> >> Can you provide more input on the use case? It is certainly possible
> to
> >> >> add support for this.
> >> >>
> >> >> If the events don't need to be inline with the records, we can easily
> >> >> setup the TaskEventDispatcher as a separate actor (or extend the task
> >> >> manager) to process both backwards flowing events and in general any
> >> events
> >> >> that don't need to be inline with the records. The task deployment
> >> >> descriptors need to be extended with the extra parallel instance
> >> >> information.
> >> >>
> >> >> – Ufuk
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message