spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-20580) Allow RDD cache with unserializable objects
Date Mon, 08 May 2017 08:03:04 GMT


Sean Owen commented on SPARK-20580:

That error isn't 'unserializable'. It's still not clear here the use case for caching unserializable
objects, nor what your code is doing.

> Allow RDD cache with unserializable objects
> -------------------------------------------
>                 Key: SPARK-20580
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.3.0
>            Reporter: Fernando Pereira
>            Priority: Minor
> In my current scenario we load complex Python objects in the worker nodes that are not
completely serializable. We then apply map certain operations to the RDD which at some point
we collect. In this basic usage all works well.
> However, if we cache() the RDD (which defaults to memory) suddenly it fails to execute
the transformations after the caching step. Apparently caching serializes the RDD data and
deserializes it whenever more transformations are required.
> It would be nice to avoid serialization of the objects if they are to be cached to memory,
and keep the original object

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message