spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravinder Matte (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated
Date Thu, 01 Feb 2018 06:15:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravinder Matte updated SPARK-23294:
-----------------------------------
    Fix Version/s:     (was: 2.2.0)

> Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-23294
>                 URL: https://issues.apache.org/jira/browse/SPARK-23294
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.2.1
>            Reporter: Ravinder Matte
>            Assignee: Takeshi Yamamuro
>            Priority: Minor
>              Labels: ConsoleSink, RateSource
>
> I am calling spark-submit passing maxRate, I have a single kinesis receiver, and batches
of 1s
> spark-submit  --conf spark.streaming.receiver.maxRate=10 ....
> however a single batch can greatly exceed the stablished maxRate. i.e: Im getting 300
records.
> it looks like Kinesis is completely ignoring the spark.streaming.receiver.maxRate configuration.
> If you look inside KinesisReceiver.onStart, you see:
> val kinesisClientLibConfiguration =
>   new KinesisClientLibConfiguration(checkpointAppName, streamName, awsCredProvider, workerId)
>   .withKinesisEndpoint(endpointUrl)
>   .withInitialPositionInStream(initialPositionInStream)
>   .withTaskBackoffTimeMillis(500)
>   .withRegionName(regionName)
> This constructor ends up calling another constructor which has a lot of default values
for the configuration. One of those values is DEFAULT_MAX_RECORDS which is constantly set
to 10,000 records.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message