Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BFF5D200B49 for ; Wed, 3 Aug 2016 18:11:29 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BEB73160A86; Wed, 3 Aug 2016 16:11:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E0B10160A5D for ; Wed, 3 Aug 2016 18:11:28 +0200 (CEST) Received: (qmail 78316 invoked by uid 500); 3 Aug 2016 16:11:28 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 78306 invoked by uid 99); 3 Aug 2016 16:11:28 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Aug 2016 16:11:28 +0000 Received: from mail-it0-f46.google.com (mail-it0-f46.google.com [209.85.214.46]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id BE0121A01AC for ; Wed, 3 Aug 2016 16:11:27 +0000 (UTC) Received: by mail-it0-f46.google.com with SMTP id u186so301988084ita.0 for ; Wed, 03 Aug 2016 09:11:27 -0700 (PDT) X-Gm-Message-State: AEkoous7iPxJpAzuo1/ZBLOhc5ZjbRj1HC36q02H9Vo86iPgV2rC4Ic2BU4F8cqDO4HW/iRQDlIh7U65fU21gg== X-Received: by 10.36.53.72 with SMTP id k69mr27317608ita.66.1470240686993; Wed, 03 Aug 2016 09:11:26 -0700 (PDT) MIME-Version: 1.0 References: <86D4F890-5419-4D90-BF41-D5F1E18E6C27@eniro.com> <5EB15A9BC333DE05.cc5078fb-c4a8-4ab1-b4d0-22ecadf01490@mail.outlook.com> In-Reply-To: <5EB15A9BC333DE05.cc5078fb-c4a8-4ab1-b4d0-22ecadf01490@mail.outlook.com> From: Aljoscha Krettek Date: Wed, 03 Aug 2016 16:11:16 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Parallel execution on AllWindows To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a114ab000e2099805392d159e archived-at: Wed, 03 Aug 2016 16:11:29 -0000 --001a114ab000e2099805392d159e Content-Type: text/plain; charset=UTF-8 Hi, "rebalance" simply specifies the strategy to use when sending elements downstream to the next operator(s). There is no interaction or competition between the parallel window operator instances. Each will do windowing locally based on the elements that it receives from upstream. Cheers, Aljoscha On Wed, 3 Aug 2016 at 08:26 wrote: > Hi Aljoscha > > Thanks for the explanation. > One other thing, when you say there is no coordination is that means > rebalance() will not be honored, and each window operator instance will > compete for the next available window? > > Thanks > > Andrew > From mobile > > From: Aljoscha Krettek > Sent: Wednesday, August 3, 17:11 > Subject: Re: Parallel execution on AllWindows > To: user@flink.apache.org > > Hi, > > if you manually force a parallelism different from 1 after a *windowAll() > then you will get parallel execution of your window. For example, if you do > this: > > input.countWindowAll(100).setParallelism(5) > > then you will get five parallel instances of the window operator that each > wait for 100 elements before they fire the window. There is no global > coordination between the parallel instances that would allow it to fire > once 100 elements are received across the parallel instances. > > Cheers, > > Aljoscha > > On Wed, 3 Aug 2016 at 05:10 Andrew Ge Wu wrote: > > Hi, > > I have such task that I want to count window on a stream and execute them > batch by batch. > > Execute a count window may take some time, so I want it to be executed in > *parallel*. > > I read this part in the documentation when I found it automatically > reduced parallelization to 1 > > * Note: This operation can be inherently non-parallel since all elements > have to pass through > * the same operator instance. (Only for special cases, such as aligned > time windows is > * it possible to perform this operation in parallel). > > (It looks like the java doc is copied from timeWindowAll) > > If I force all window function to run in parallel, what will happen? > > Will a time/count window broadcast to all instances of the function? or > will it be send to one of the instance so I can parallelize my work? > > Thanks! > > > Andrew > > Confidentiality Notice: This e-mail transmission may contain confidential > or legally privileged information that is intended only for the individual > or entity named in the e-mail address. If you are not the intended > recipient, you are hereby notified that any disclosure, copying, > distribution, or reliance upon the contents of this e-mail is strictly > prohibited and may be unlawful. If you have received this e-mail in error, > please notify the sender immediately by return e-mail and delete all copies > of this message. > > > > Confidentiality Notice: This e-mail transmission may contain confidential > or legally privileged information that is intended only for the individual > or entity named in the e-mail address. If you are not the intended > recipient, you are hereby notified that any disclosure, copying, > distribution, or reliance upon the contents of this e-mail is strictly > prohibited and may be unlawful. If you have received this e-mail in error, > please notify the sender immediately by return e-mail and delete all copies > of this message. --001a114ab000e2099805392d159e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
"rebalance" simply specifies the strateg= y to use when sending elements downstream to the next operator(s). There is= no interaction or competition between the parallel window operator instanc= es. Each will do windowing locally based on the elements that it receives f= rom upstream.

Cheers,
Aljoscha

On Wed, 3 Aug 2016 at 08:= 26 <andrew.ge-wu@eniro.com= > wrote:

Hi Aljo= scha

Thanks for the explanation.
One other thing, when you say there is no coordination is that means rebala= nce() will not be honored, and each window operator instance will compete f= or the next available window?

Thanks

Andrew
From mobile

From: Aljoscha Krettek
Sent: Wednesday, August 3, 17:11
Subject: Re: Parallel execution on AllWindows
To: user@flink.a= pache.org

Hi,

if you manually force a parallelism different from 1 after = a *windowAll() then you will get parallel execution of your window. For exa= mple, if you do this:

input.countWindowAll(100).setParallelism(5)

then you will get five parallel instances of the window ope= rator that each wait for 100 elements before they fire the window. There is= no global coordination between the parallel instances that would allow it = to fire once 100 elements are received across the parallel instances.

Cheers,

Aljoscha

On Wed, 3 Aug 2016 at 05:10 Andrew Ge Wu <andrew.ge-wu@eniro.com>= ; wrote:

Hi,

I have such task that I want to count window on a stream an= d execute them batch by batch.

Execute a count window may take some time, so I want it to = be executed in parallel.

I read this part in the documentation when I found it autom= atically reduced parallelization to 1

* Note: This operation can be inherently non-parallel since= all elements have to pass through
* the same operator instance. (Only for special cases, such as aligned time= windows is
* it possible to perform this operation in parallel).

(It looks like the java doc is copied from timeWindowAll)

If I force all window function to run in parallel, what wil= l happen?

Will a time/count window broadcast to all instances of the = function? or will it be send to one of the instance so I can parallelize my= work?

Thanks!


Andrew

Confidentiality Notice: This e-mail transmission may contai= n confidential or legally privileged information that is intended only for = the individual or entity named in the e-mail address. If you are not the in= tended recipient, you are hereby notified that any disclosure, copying, dis= tribution, or reliance upon the contents of this e-mail is strictly prohibi= ted and may be unlawful. If you have received this e-mail in error, please = notify the sender immediately by return e-mail and delete all copies of thi= s message.



Confidentiality Notice: This e-ma= il transmission may contain confidential or legally privileged information = that is intended only for the individual or entity named in the e-mail addr= ess. If you are not the intended recipient, you are hereby notified that an= y disclosure, copying, distribution, or reliance upon the contents of this = e-mail is strictly prohibited and may be unlawful. If you have received thi= s e-mail in error, please notify the sender immediately by return e-mail an= d delete all copies of this message.
--001a114ab000e2099805392d159e--