flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sameer W <sam...@axiomine.com>
Subject Re: Multiple Partitions (Source Functions) -> Event Time -> Watermarks -> Trigger
Date Wed, 10 Aug 2016 19:27:45 GMT
And this is happening in my local environment. As soon as I set the
parallelism to 1 it all works fine.

Sameer

On Wed, Aug 10, 2016 at 3:11 PM, Sameer W <sameer@axiomine.com> wrote:

> Hi,
>
> I am noticing this behavior with Event Time processing-
>
> I have a Kafka topic with 10 partitions. Each Event Source sends data to
> any one of the partitions. Say I have only 1 event source active at this
> moment, which means only one partition is receiving data.
>
> None of my windows will fire now because the 9 partitions (source function
> instances) are not sending any watermarks and Flink waits forever.
>
> I go to topic with 1 partition but leave default parallelism intact. Only
> one Mapper instance contributes to the subsequent keyBy operation but other
> 7 (assuming 8 of default parallelism) are idle. I assign watermarks after
> the map function. Again the same behavior because the 7 other mappers are
> not sending watermarks.
>
> How do I handle this? Not all of my partitions are going to be receiving
> data at all times using this partitioning strategy. Or I have to use random
> partitioning which will also work.
>
> Thanks,
> Sameer
>

Mime
View raw message