flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aggarwal, Ajay" <Ajay.Aggar...@netapp.com>
Subject Impact of occasional big pauses in stream processing
Date Wed, 13 Feb 2019 16:50:34 GMT
I was wondering what is the impact if one of the stream operator function occasionally takes
too long to process the event.  Given the following simple flink job

       inputStream
          .KeyBy (“tenantId”)
          .process ( new MyKeyedProcessFunction())

, if occasionally MyKeyedProcessFunction takes too long (say ~5-10 minutes) to process an
incoming element, what is the impact on overall pipeline? Is the impact limited to

  1.  Specific key for which MyKeyedProcessFunction is currently taking too long to process
an element, or
  2.  Specific Taskslot, where MyKeyedProcessFunction is currently taking too long to process
an element, i.e. impacting multiple keys, or
  3.  Entire inputstream ?

Also what is the built in resiliency in these cases? Is there a concept of timeout for each
operator function?

Ajay
Mime
View raw message