Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@avro.apache.org
Date: Fri, 22 Mar 2013 21:43:15 +0000 (UTC)
From: "Leo Romanoff (JIRA)" <jira@apache.org>
To: dev@avro.apache.org
Message-ID: <JIRA.12638536.1363971445284.39912.1363988595668@arcas>
In-Reply-To: <JIRA.12638536.1363971445284@arcas>
References: <JIRA.12638536.1363971445284@arcas>
Subject: [jira] [Commented] (AVRO-1282) Make use of the sun.misc.Unsafe
 class during serialization if a JDK supports it
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611299#comment-13611299 ] 

Leo Romanoff commented on AVRO-1282:
------------------------------------

> I'm hesitant to add native-byte-order serialization to Avro. A primary purpose for Avro is as an interchange format. 
> Native byte-order would be an incompatible change.

I suspected this reaction ;-) Let's postpone this feature it for now. IMHO, having a native byte order is not bad as such, as long as you indicate in the produced representation which concrete byte order was used during serialization (e.g. big endian, little endian, etc). This can be an attribute or something like this. If this is provided, the other side can always deserialize without problems, even if it uses a different byte order. In this case it would simply apply a conversion. And if it has the same byte-order it can deserialize at full speed. I think for Big Data applications which use Avro as internal storage format (are there such apps/frameworks, e.g. Voldermont?), it could be a huge win, because you usually use the same CPU architecture in your data center, or?

> Improving reflection performance would be a clear win.

OK. I will concentrate on this fist.
                
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
> -------------------------------------------------------------------------------
>
>                 Key: AVRO-1282
>                 URL: https://issues.apache.org/jira/browse/AVRO-1282
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.4
>            Reporter: Leo Romanoff
>            Priority: Minor
>
> Unsafe can be used to significantly speed up serialization process, if a JDK implementation supports java.misc.Unsafe properly. Most JDKs running on PCs support it. Some platforms like Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the reflection-based approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick input/output.
>  
> 3) More over, Unsafe makes it possible to serialize to/deserialize from off-heap memory directly and very quickly, without any intermediate buffers allocated on heap. There is virtually no overhead compared to the usual byte arrays.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira