avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Romanoff (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1282) Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
Date Fri, 22 Mar 2013 21:43:15 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611299#comment-13611299
] 

Leo Romanoff commented on AVRO-1282:
------------------------------------

> I'm hesitant to add native-byte-order serialization to Avro. A primary purpose for Avro
is as an interchange format. 
> Native byte-order would be an incompatible change.

I suspected this reaction ;-) Let's postpone this feature it for now. IMHO, having a native
byte order is not bad as such, as long as you indicate in the produced representation which
concrete byte order was used during serialization (e.g. big endian, little endian, etc). This
can be an attribute or something like this. If this is provided, the other side can always
deserialize without problems, even if it uses a different byte order. In this case it would
simply apply a conversion. And if it has the same byte-order it can deserialize at full speed.
I think for Big Data applications which use Avro as internal storage format (are there such
apps/frameworks, e.g. Voldermont?), it could be a huge win, because you usually use the same
CPU architecture in your data center, or?

> Improving reflection performance would be a clear win.

OK. I will concentrate on this fist.
                
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
> -------------------------------------------------------------------------------
>
>                 Key: AVRO-1282
>                 URL: https://issues.apache.org/jira/browse/AVRO-1282
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.4
>            Reporter: Leo Romanoff
>            Priority: Minor
>
> Unsafe can be used to significantly speed up serialization process, if a JDK implementation
supports java.misc.Unsafe properly. Most JDKs running on PCs support it. Some platforms like
Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the reflection-based
approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick input/output.
>  
> 3) More over, Unsafe makes it possible to serialize to/deserialize from off-heap memory
directly and very quickly, without any intermediate buffers allocated on heap. There is virtually
no overhead compared to the usual byte arrays.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message