Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BE3DA200B8B for ; Tue, 4 Oct 2016 11:45:34 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BCACD160AC9; Tue, 4 Oct 2016 09:45:34 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 058CC160AC5 for ; Tue, 4 Oct 2016 11:45:33 +0200 (CEST) Received: (qmail 29033 invoked by uid 500); 4 Oct 2016 09:45:33 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 29024 invoked by uid 99); 4 Oct 2016 09:45:33 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Oct 2016 09:45:33 +0000 Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 5A49D1A0118 for ; Tue, 4 Oct 2016 09:45:32 +0000 (UTC) Received: by mail-wm0-f54.google.com with SMTP id b201so133325335wmb.0 for ; Tue, 04 Oct 2016 02:45:32 -0700 (PDT) X-Gm-Message-State: AA6/9RlZVZ1iHdmSYmm3e9ePk+rtwu7WL89GP6eaYfNRWBHrqY5Yna6IaIhGNGgr+AQNHHjF4yyU6jsBBiOVng== X-Received: by 10.28.141.133 with SMTP id p127mr14427984wmd.119.1475574330851; Tue, 04 Oct 2016 02:45:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.216.71 with HTTP; Tue, 4 Oct 2016 02:45:30 -0700 (PDT) In-Reply-To: References: From: Till Rohrmann Date: Tue, 4 Oct 2016 11:45:30 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Side Inputs vs. Connected Streams To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a11474226d48dbc053e06eb61 archived-at: Tue, 04 Oct 2016 09:45:34 -0000 --001a11474226d48dbc053e06eb61 Content-Type: text/plain; charset=UTF-8 Hi Sameer, the semantics of side inputs are not fully fledged out yet, as far as I know. The design doc says that one first waits for some data on the side input before starting processing the main input. Therefore, I would assume that you have some temporal ordering of the side input and main input compared to connected streams. But don't take this as guaranteed since this feature is still under development and the semantics are not fully decided yet. Cheers, Till On Mon, Oct 3, 2016 at 2:31 PM, Sameer W wrote: > Hi, > > I read the Side Inputs > > design document. How does it compare to using ConnectedStreams with respect > to handling the ordering of streams transparently? > > One of the challenges I have with ConnectedStreams is I need to buffer > main input if the rules stream has not arrived yet. Does this automatically > go away with Side Inputs? Will the call to String sideValue = > > getRuntimeContext().getSideInput(filterString); > block if the side input is not available yet? And is the reverse also true? > > Alternatively, if my rules are not large in number and I want to broadcast > them to all nodes is the below equivalent to using SideInputs where side > inputs are broadcast to all nodes and ensure that the side input is > evaluated before the main input: > > DataStream ds4 = ds3.connect(dsSide.broadcast()); > > Will the above ensure that dsSide is always available before ds3 elements > arrive on the connected stream. Am I correct in assuming that ds2 changes > will continue to be broadcast to ds3 (with no ordering guarantees between > ds3 and dsSide, ofcourse). > > > Thanks, > Sameer > > > > --001a11474226d48dbc053e06eb61 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Sameer,

the semantics of side inputs= are not fully fledged out yet, as far as I know.

= The design doc says that one first waits for some data on the side input be= fore starting processing the main input. Therefore, I would assume that you= have some temporal ordering of the side input and main input compared to c= onnected streams.=C2=A0

But don't take this as= guaranteed since this feature is still under development and the semantics= are not fully decided yet.

Cheers,
Till=

On Mo= n, Oct 3, 2016 at 2:31 PM, Sameer W <sameer@axiomine.com> = wrote:
Hi,

I read the Side Inputs d= esign document. How does it compare to using ConnectedStreams with respect = to handling the ordering of streams transparently?=C2=A0

One of the challenges I have with ConnectedStreams is I need to buff= er main input if the rules stream has not arrived yet. Does this automatica= lly go away with Side Inputs? Will the call to=C2=A0 String sideValue =3D

= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0getRuntimeContext().getSideI= nput(filterString);

block if the side input is not av= ailable yet? And is the reverse also true?

Alt= ernatively, if my rules are not large in number and I want to broadcast the= m to all nodes is the below equivalent to using SideInputs where side input= s are broadcast to all nodes and ensure that the side input is evaluated be= fore the main input:

DataStream ds4 =3D ds3.connect(dsSide.broadcast());

Will the above ensure that dsSide is always available befor= e ds3 elements arrive on the connected stream. Am I correct in assuming tha= t ds2 changes will continue to be broadcast to ds3 (with no ordering guaran= tees between ds3 and dsSide, ofcourse).=


Thanks,
Samee= r




--001a11474226d48dbc053e06eb61--