avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1282) Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
Date Tue, 23 Apr 2013 18:15:17 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639339#comment-13639339
] 

Scott Carey commented on AVRO-1282:
-----------------------------------

Yes, ResolvingDecoder is a bottleneck.  I have several ideas for abolishing it completely
but they aren't trivial.  In general, decoding involves continuously traversing multiple data
structures -- the Parser and the object graph being built or read, as well as the Schema.
 Instead, building the read pipeline and resolving by composition of small functional bits
and precomputing all of the possibilities into a composite function will be faster.  This
composite function can then pass through ASM to 'devirtualize' it and inline some operations.
 That is out of scope for this ticket, but enhancing what we can in the current framework
is not a bad idea.

Re:  Float/Double
How much do those improvements help the Generic tests? I assume they are still dominated by
the issues with the decoder.  Although decreasing the time of a Float read by 35% is a big
win, it is a small part of what happens with Generic/Specific/Reflect reading.  Does using
unsafe here break anything if the system's byte order does not match?

Your results for reading Floats/Doubles above are slower than mine, is this because you are
now populating an array when reading as well?  The benchmark was intended to isolate (as best
as possible) the read and write portions of the decoder/encoder?  I'd be interested in looking
at this code.


As for arrays of primitives, we should look into Group varint encoding and similar techniques:
http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/people/jeff/WSDM09-keynote.pdf
 (go to page 55).


In general, lets separate out the Unsafe enhancements for Reflection from those that are for
the Input/Output streams.  Lets get the reflection work in this ticket done and committed,
then move on to other enhancements.


We have three tickets then:
* This one related to reflection improvements via Unsafe.
* Another one related  to input/output improvements via Unsafe.
* A third related to other non-Unsafe performance improvements, for example those you allude
to with:
{quote}
Coming back to your message about reading doubles and floats and observation that they are
much slower with ReflectDatumReader/Writer. I just did some profiling and found some issues.
With small changes, I could significantly improve write performance and improved a bit read
performance. All this without Unsafe reads/writes into streams yet.
{quote}

I think we should try and separate these from each other.

                
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
> -------------------------------------------------------------------------------
>
>                 Key: AVRO-1282
>                 URL: https://issues.apache.org/jira/browse/AVRO-1282
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.4
>            Reporter: Leo Romanoff
>            Priority: Minor
>         Attachments: avro-1282-v1.patch, avro-1282-v2.patch, avro-1282-v3.patch, avro-1282-v4.patch,
avro-1282-v5.patch, avro-1282-v6.patch
>
>
> Unsafe can be used to significantly speed up serialization process, if a JDK implementation
supports java.misc.Unsafe properly. Most JDKs running on PCs support it. Some platforms like
Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the reflection-based
approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick input/output.
>  
> 3) More over, Unsafe makes it possible to serialize to/deserialize from off-heap memory
directly and very quickly, without any intermediate buffers allocated on heap. There is virtually
no overhead compared to the usual byte arrays.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message