spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Heji Kim (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-11613) Kinesis ASL should allow caller to set ClientConfiguration for socket timeouts and other connection settings
Date Tue, 24 Nov 2015 02:42:11 GMT

     [ https://issues.apache.org/jira/browse/SPARK-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Heji Kim updated SPARK-11613:
-----------------------------
    Description: 
Currently the Kinesis ASL does not allow configuration of connection settings on AmazonKinesisClient
such as the socket timeout or the maxRecords fetched per getRecords.

Currently we get intermittent 2 minute SocketTimeoutExceptions from Kinesis which holds up
the driver processing and pile up batch processing. 
We'd like to set it to something closer to the batch polling time of 5 seconds. The requests
are normally subsecond so we'd rather fail fast.

KinesisBackedBlockRDD.KinesisSequenceRangeIterator uses only the credential based API to create
AmazonKinesisClient and so we don't have access to com.amazonaws.ClientConfiguration which
is the only API AWS provides for setting connection settings.

  was:
Currently the Kinesis ASL does not allow configuration of connection settings on AmazonKinesisClient
such as the socket timeout.

Currently we get intermittent 2 minute SocketTimeoutExceptions from Kinesis which holds up
the driver processing and pile up batch processing. We'd like to set it to something closer
to the batch polling time of 5 seconds. The requests are normally subsecond so we'd rather
fail fast.

KinesisBackedBlockRDD.KinesisSequenceRangeIterator uses only the credential based API to create
AmazonKinesisClient and so we don't have access to com.amazonaws.ClientConfiguration which
is the only API AWS provides for setting connection settings.


> Kinesis ASL should allow caller to set ClientConfiguration for socket timeouts and other
connection settings
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-11613
>                 URL: https://issues.apache.org/jira/browse/SPARK-11613
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.5.1
>            Reporter: Heji Kim
>
> Currently the Kinesis ASL does not allow configuration of connection settings on AmazonKinesisClient
such as the socket timeout or the maxRecords fetched per getRecords.
> Currently we get intermittent 2 minute SocketTimeoutExceptions from Kinesis which holds
up the driver processing and pile up batch processing. 
> We'd like to set it to something closer to the batch polling time of 5 seconds. The requests
are normally subsecond so we'd rather fail fast.
> KinesisBackedBlockRDD.KinesisSequenceRangeIterator uses only the credential based API
to create AmazonKinesisClient and so we don't have access to com.amazonaws.ClientConfiguration
which is the only API AWS provides for setting connection settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message