flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Parallel execution on AllWindows
Date Wed, 03 Aug 2016 15:11:21 GMT
Hi,
if you manually force a parallelism different from 1 after a *windowAll()
then you will get parallel execution of your window. For example, if you do
this:

input.countWindowAll(100).setParallelism(5)

then you will get five parallel instances of the window operator that each
wait for 100 elements before they fire the window. There is no global
coordination between the parallel instances that would allow it to fire
once 100 elements are received across the parallel instances.

Cheers,
Aljoscha

On Wed, 3 Aug 2016 at 05:10 Andrew Ge Wu <andrew.ge-wu@eniro.com> wrote:

> Hi,
>
> I have such task that I want to count window on a stream and execute them
> batch by batch.
> Execute a count window may take some time, so I want it to be executed in
> *parallel*.
> I read this part in the documentation when I found it automatically
> reduced parallelization to 1
>
> * Note: This operation can be inherently non-parallel since all elements have to pass
through
> * the same operator instance. (Only for special cases, such as aligned time windows is
> * it possible to perform this operation in parallel).
>
> (It looks like the java doc is copied from timeWindowAll)
>
> If I force all window function to run in parallel, what will happen?
> Will a time/count window broadcast to all instances of the function? or will it be send
to one of the instance so I can parallelize my work?
>
>
> Thanks!
>
>
>
> Andrew
>
>
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.

Mime
View raw message