spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yash Sharma (JIRA)" <>
Subject [jira] [Created] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries
Date Wed, 29 Mar 2017 12:22:41 GMT
Yash Sharma created SPARK-20140:

             Summary: Remove hardcoded kinesis retry wait and max retries
                 Key: SPARK-20140
             Project: Spark
          Issue Type: Bug
          Components: DStreams
    Affects Versions: 2.1.0
            Reporter: Yash Sharma

The pull requests proposes to remove the hardcoded values for Amazon Kinesis - MIN_RETRY_WAIT_TIME_MS,

This change is critical for kinesis checkpoint recovery when the kinesis backed rdd is huge.
Following happens in a typical kinesis recovery :
- kinesis throttles large number of requests while recovering
- retries in case of throttling are not able to recover due to the small wait period
- kinesis throttles per second, the wait period should be configurable for recovery

The patch picks the spark kinesis configs from:
- spark.streaming.kinesis.retry.wait.time
- spark.streaming.kinesis.retry.max.attempts

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message