avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Romanoff (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1282) Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
Date Sun, 21 Apr 2013 08:47:16 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637508#comment-13637508
] 

Leo Romanoff commented on AVRO-1282:
------------------------------------

@Doug: Thanks that you confirmed my measurements! 3x is nice indeed. On my MacBook Air with
i7 and 8 GB of memory I've seen even 4-5 speedups on the same tests. So, it seems to be a
big performance win in any case. Do you plan to include it into the trunk? If so, when?

Coming back to the beginning of this thread:
Improvements we have seen so far are purely due to the optimization of accesses to the fields
of structures and avoiding reflection as much as possible. 
Based on my experience with Kryo, it could be even more beneficial to use Unsafe-based approach
for Input/Output streams. But this would affect the format of produced data, because Unsafe
reads/writes primitive types using their native size and native byte order. We touched a bit
upon this at the beginning, but may be it is time now to decide what to do with it? Do you
see any option to support it in Avro, e.g. by introducing a new format id or something like
this? or may be by introducing new type ids (something like LITTLE_ENDIAN_INT, LITTLE_ENDIAN_LONG,
LITTLE_ENDIAN_FLOAT, LITTLE_ENDIAN_DOUBLE,etc and the same for BIG_ENDIAN). This way one could
use both encodings for different parts/fields of the same object grpah. What do you think?




 


                
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
> -------------------------------------------------------------------------------
>
>                 Key: AVRO-1282
>                 URL: https://issues.apache.org/jira/browse/AVRO-1282
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.4
>            Reporter: Leo Romanoff
>            Priority: Minor
>         Attachments: avro-1282-v1.patch, avro-1282-v2.patch, avro-1282-v3.patch, avro-1282-v4.patch,
avro-1282-v5.patch, avro-1282-v6.patch
>
>
> Unsafe can be used to significantly speed up serialization process, if a JDK implementation
supports java.misc.Unsafe properly. Most JDKs running on PCs support it. Some platforms like
Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the reflection-based
approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick input/output.
>  
> 3) More over, Unsafe makes it possible to serialize to/deserialize from off-heap memory
directly and very quickly, without any intermediate buffers allocated on heap. There is virtually
no overhead compared to the usual byte arrays.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message