spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <>
Subject [jira] [Updated] (SPARK-1811) Support resizable output buffer for kryo serializer
Date Thu, 29 May 2014 06:26:01 GMT


Matei Zaharia updated SPARK-1811:

    Assignee: Koert Kuipers

> Support resizable output buffer for kryo serializer
> ---------------------------------------------------
>                 Key: SPARK-1811
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: koert kuipers
>            Assignee: Koert Kuipers
>            Priority: Minor
> Currently the size of kryo serializer output buffer can be set with spark.kryoserializer.buffer.mb
> The issue with this setting is that it has to be one-size-fits-all, so it ends up being
the maximum size needed, even if only a single task out of many needs it to be that big. A
resizable buffer will allow most tasks to use a modest sized buffer while the incidental task
that needs a really big buffer can get it at a cost (allocating a new buffer and copying the
contents over repeatedly as the buffer grows... with each new allocation the size doubles).
> The class used for the buffer is kryo Output, which supports resizing if  maxCapacity
is set bigger than capacity. I suggest we provide a setting spark.kryoserializer.buffer.max.mb
which defaults to spark.kryoserializer.buffer.mb, and which sets Output's maxCapacity.
> Pull request for this jira:

This message was sent by Atlassian JIRA

View raw message