flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Qin <qinnc...@gmail.com>
Subject large sliding window perf question
Date Tue, 23 May 2017 19:24:14 GMT
Hi there,

I have seen some weird perf issue while running event time based job with
large sliding window (24 hours offset every 10s)

pipeline looks simple,
tail kafka topic and assign timestamp and watermark, forward to large
sliding window (30days) and fire every 10 seconds and print out.

what I have seen first hand was checkpointing stuck, took longer than
timeout despite traffic volume is low ~300 TPS. Looking deeper, it seems
back pressure kick in and window operator consumes message really slowly
and throttle sources.

I also tried to limit window time to mins and all issues are gone.

Any suggestion on this. My work around is I implemented processFunction and
keep big value state, periodically evaluate and emit downstream (emulate
what sliding window does)

Thanks,
Chen

Mime
View raw message