spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandeep Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-928) Add support for Unsafe-based serializer in Kryo 2.22
Date Sun, 01 May 2016 16:15:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265805#comment-15265805
] 

Sandeep Singh commented on SPARK-928:
-------------------------------------

[~joshrosen] I've started working on it.

I tried benchmarking the difference between unsafe kryo and our current impl. and then we
can have a spark.kryo.useUnsafe flag as Matei has mentioned.

{code:title=Without Kryo UnSafe|borderStyle=solid}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz

Serialize and then deserialize:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------
primitive:Long                              1 /    4      11223.1           0.1       1.0X
primitive:Double                            1 /    1      19409.0           0.1       1.7X
Array:Long                                 38 /   49        412.4           2.4       0.0X
Array:Double                               25 /   35        631.4           1.6       0.1X
Map of string->Double                    2651 / 2766          5.9         168.6       0.0X
{code}

{code:title=With Kryo UnSafe|borderStyle=solid}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz

Serialize and then deserialize:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------
primitive:Long                              1 /    3      15872.0           0.1       1.0X
primitive:Double                            1 /    1      17769.7           0.1       1.1X
Array:Long                                 24 /   42        642.3           1.6       0.0X
Array:Double                               22 /   26        719.4           1.4       0.0X
Map of string->Double                    2560 / 2582          6.1         162.8       0.0X
{code}

You can find the code for benchmarking here (https://github.com/techaddict/spark/commit/46fa44141c849ca15bbd6136cea2fa52bd927da2),
very ugly right now but will improve it(add more benchmarks) before creating a PR.


> Add support for Unsafe-based serializer in Kryo 2.22
> ----------------------------------------------------
>
>                 Key: SPARK-928
>                 URL: https://issues.apache.org/jira/browse/SPARK-928
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Matei Zaharia
>            Priority: Minor
>              Labels: starter
>
> This can reportedly be quite a bit faster, but it also requires Chill to update its Kryo
dependency. Once that happens we should add a spark.kryo.useUnsafe flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message