spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From swetha <swethakasire...@gmail.com>
Subject Re: Does RDD checkpointing store the entire state in HDFS?
Date Tue, 14 Jul 2015 23:11:34 GMT

Hi TD, 

I have a question regarding sessionization using updateStateByKey. If near
real time state needs to be maintained in a Streaming application, what
happens when the number of RDDs to maintain the state becomes very large?
Does it automatically get saved to HDFS and reload when needed or do I have
to use any code like ssc.checkpoint(checkpointDir)?  Also, how is the
performance if I use both DStream Checkpointing for maintaining the state
and use Kafka Direct approach for exactly once semantics?


Thanks, 
Swetha



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Does-RDD-checkpointing-store-the-entire-state-in-HDFS-tp7368p13227.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message