flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Side Inputs vs. Connected Streams
Date Tue, 04 Oct 2016 09:45:30 GMT
Hi Sameer,

the semantics of side inputs are not fully fledged out yet, as far as I
know.

The design doc says that one first waits for some data on the side input
before starting processing the main input. Therefore, I would assume that
you have some temporal ordering of the side input and main input compared
to connected streams.

But don't take this as guaranteed since this feature is still under
development and the semantics are not fully decided yet.

Cheers,
Till

On Mon, Oct 3, 2016 at 2:31 PM, Sameer W <sameer@axiomine.com> wrote:

> Hi,
>
> I read the Side Inputs
> <https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit#>
> design document. How does it compare to using ConnectedStreams with respect
> to handling the ordering of streams transparently?
>
> One of the challenges I have with ConnectedStreams is I need to buffer
> main input if the rules stream has not arrived yet. Does this automatically
> go away with Side Inputs? Will the call to  String sideValue =
>
>        getRuntimeContext().getSideInput(filterString);
> block if the side input is not available yet? And is the reverse also true?
>
> Alternatively, if my rules are not large in number and I want to broadcast
> them to all nodes is the below equivalent to using SideInputs where side
> inputs are broadcast to all nodes and ensure that the side input is
> evaluated before the main input:
>
> DataStream ds4 = ds3.connect(dsSide.broadcast());
>
> Will the above ensure that dsSide is always available before ds3 elements
> arrive on the connected stream. Am I correct in assuming that ds2 changes
> will continue to be broadcast to ds3 (with no ordering guarantees between
> ds3 and dsSide, ofcourse).
>
>
> Thanks,
> Sameer
>
>
>
>

Mime
View raw message