spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Spark Newbie <>
Subject Re: Error in block pushing thread puts the KinesisReceiver in a stuck state
Date Mon, 30 Nov 2015 18:39:21 GMT
Pinging again to see if anyone has any thoughts or prior experience with
this issue.

On Wed, Nov 25, 2015 at 3:56 PM, Spark Newbie <>

> Hi Spark users,
> I have been seeing this issue where receivers enter a "stuck" state after
> it encounters a the following exception "Error in block pushing thread -
> java.util.concurrent.TimeoutException: Futures timed out".
> I am running the application on spark-1.4.1 and using kinesis-asl-1.4.
> When this happens, the observation is that the
> Kinesis.ProcessTask.shardxxxx.MillisBehindLatest metric does not get
> published anymore, when I look at cloudwatch, which indicates that the
> workers associated with the receiver are not checkpointing any more for the
> shards that they were reading from.
> This seems like a bug in to BlockGenerator code , here -
> when pushBlock encounters an exception, in this case the TimeoutException,
> it stops pushing blocks. Is this really expected behavior?
> Has anyone else seen this error and have you also seen the issue where
> receivers stop receiving records? I'm also trying to find the root cause
> for the TimeoutException. If anyone has an idea on this please share.
> Thanks,
> Bharath

View raw message