spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject How StorageLevel, CacheManager and checkpointing influence computing RDD partitions?
Date Sat, 10 Oct 2015 14:37:39 GMT

I've been reviewing the Spark code and noticed that `iterator` method
of RDD [1] does a check whether RDD has a non-NONE storage and calls
`computeOrReadCheckpoint` private method [2] that checks RDD

Is there a doc on how StorageLevel, CacheManager and checkpointing
influence partition computation?

Specifically, why would I have NONE StorageLevel and RDD checkpointing
enabled? What is the use case for such a configuration? What about the
other options?

Any pointers are greatly appreciated, including blog posts,
StackOverflow, Quora, archive.



Jacek Laskowski | |
Follow me at
Upvote at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message