flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-7622) Respect local KPL queue size in FlinkKinesisProducer when adding records to KPL client
Date Thu, 14 Sep 2017 12:36:00 GMT
Tzu-Li (Gordon) Tai created FLINK-7622:
------------------------------------------

             Summary: Respect local KPL queue size in FlinkKinesisProducer when adding records
to KPL client
                 Key: FLINK-7622
                 URL: https://issues.apache.org/jira/browse/FLINK-7622
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
            Reporter: Tzu-Li (Gordon) Tai


This issue was brought to discussion by [~sthm] offline.

Currently, records are added to the Kinesis KPL producer client without checking the number
of outstanding records within the local KPL queue. This manner is basically neglecting backpressure
when producing to Kinesis through KPL, and can therefore exhaust system resources.

We should respect {{producer.getOutstandingRecordsCount()}} as a measure of backpressure,
and propagate backpressure upstream by blocking further sink invocations when some threshold
of outstanding record count is exceeded. The recommended threshold [1] seems to be 10,000.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message