spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Fregly <ch...@fregly.com>
Subject Re: Spark Streaming - What does Spark Streaming checkpoint?
Date Fri, 22 Aug 2014 01:02:28 GMT
The StreamingContext can be recreated from a checkpoint file, indeed.

check out the following Spark Streaming source files for details:
 StreamingContext, Checkpoint, DStream, DStreamCheckpoint, and DStreamGraph.




On Wed, Jul 9, 2014 at 6:11 PM, Yan Fang <yanfang724@gmail.com> wrote:

> Hi guys,
>
> I am a little confusing by the checkpointing in Spark Streaming. It
> checkpoints the intermediate data for the stateful operations for sure.
> Does it also checkpoint the information of StreamingContext? Because it
> seems we can recreate the SC from the checkpoint in a driver node failure
> scenario. When I looked at the checkpoint directory, did not find much
> clue. Any help? Thank you very much.
>
> Best,
>
> Fang, Yan
> yanfang724@gmail.com
> +1 (206) 849-4108
>

Mime
View raw message