spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1811) Support resizable output buffer for kryo serializer
Date Thu, 29 May 2014 06:26:01 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matei Zaharia updated SPARK-1811:
---------------------------------

    Assignee: Koert Kuipers

> Support resizable output buffer for kryo serializer
> ---------------------------------------------------
>
>                 Key: SPARK-1811
>                 URL: https://issues.apache.org/jira/browse/SPARK-1811
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: koert kuipers
>            Assignee: Koert Kuipers
>            Priority: Minor
>
> Currently the size of kryo serializer output buffer can be set with spark.kryoserializer.buffer.mb
> The issue with this setting is that it has to be one-size-fits-all, so it ends up being
the maximum size needed, even if only a single task out of many needs it to be that big. A
resizable buffer will allow most tasks to use a modest sized buffer while the incidental task
that needs a really big buffer can get it at a cost (allocating a new buffer and copying the
contents over repeatedly as the buffer grows... with each new allocation the size doubles).
> The class used for the buffer is kryo Output, which supports resizing if  maxCapacity
is set bigger than capacity. I suggest we provide a setting spark.kryoserializer.buffer.max.mb
which defaults to spark.kryoserializer.buffer.mb, and which sets Output's maxCapacity.
> Pull request for this jira:
> https://github.com/apache/spark/pull/735



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message