spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From swetha <>
Subject Re: Does RDD checkpointing store the entire state in HDFS?
Date Tue, 14 Jul 2015 23:11:34 GMT

Hi TD, 

I have a question regarding sessionization using updateStateByKey. If near
real time state needs to be maintained in a Streaming application, what
happens when the number of RDDs to maintain the state becomes very large?
Does it automatically get saved to HDFS and reload when needed or do I have
to use any code like ssc.checkpoint(checkpointDir)?  Also, how is the
performance if I use both DStream Checkpointing for maintaining the state
and use Kafka Direct approach for exactly once semantics?


View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message