avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AVRO-1282) Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
Date Mon, 29 Apr 2013 21:48:16 GMT

     [ https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Scott Carey updated AVRO-1282:
------------------------------

    Attachment: AVRO-1282-s6.patch

This patch (AVRO-1282-s6.patch) includes the following:

* The Reflect API now uses sun.misc.Unsafe to do Field reflection, for significantly improved
performance. (Leo)
* The Reflect API avoids boxing for reading and writing primitive arrays.  (Leo, Scott)
* Resolution of whether it is safe to use Unsafe is done statically, only one implementation
is loaded.  The Unsafe implementation is tested to ensure that all features function properly
at load time (for example, to handle Android or other JVMs with partial Unsafe support). 
A unit test is added that uses a classloader that fails to load Unsafe to test this. (Leo,
Scott)
* EncoderFactory is fixed to properly configure the BlockingBinaryEncoder. (Scott)
* Reflection now supports reading the Blocked encoding into native arrays. (Scott)
* A dozen new tests added to Perf.java to cover the ReflectDatum{Reader,Writer}, including
with the blocked binary encoding. (Leo, Scott)
* Additional unit tests in TestReflect to cover encoding and decoding primitive arrays, including
with blocked binary encoding. (Scott)

Performance on Reflect Perf.java tests is approximately 2.5x to 29x faster, usually between
3x and 5x faster.

Before:
{noformat}
                                 test name     time    M entries/sec   M bytes/sec  bytes/cycle
                         ReflectRecordRead:  15067 ms       1.106        42.927        808498
                        ReflectRecordWrite:  10903 ms       1.529        59.319        808498
                      ReflectBigRecordRead:  19285 ms       0.519        31.832        767380
                     ReflectBigRecordWrite:  13958 ms       0.716        43.979        767380
                          ReflectFloatRead:  29594 ms       0.000        27.032       1000004
                         ReflectFloatWrite:  33327 ms       0.000        24.004       1000004
                         ReflectDoubleRead:  27442 ms       0.000        58.303       2000004
                        ReflectDoubleWrite:  31529 ms       0.000        50.746       2000004
                       ReflectIntArrayRead:  38088 ms       0.438        18.057        859709
                      ReflectIntArrayWrite:  23342 ms       0.714        29.464        859709
                      ReflectLongArrayRead:  18476 ms       0.451        34.869        805344
                     ReflectLongArrayWrite:  12715 ms       0.655        50.667        805344
                    ReflectDoubleArrayRead:  19411 ms       0.515        33.718        818144
                   ReflectDoubleArrayWrite:  13825 ms       0.723        47.340        818144
                     ReflectFloatArrayRead:  39502 ms       0.506        17.137        846172
                    ReflectFloatArrayWrite:  27492 ms       0.727        24.623        846172
               ReflectNestedFloatArrayRead:  41225 ms       0.485        16.420        846172
              ReflectNestedFloatArrayWrite:  30229 ms       0.662        22.393        846172
              ReflectNestedObjectArrayRead:  31679 ms       0.126        16.291        645104
             ReflectNestedObjectArrayWrite:  17206 ms       0.232        29.994        645104
          ReflectNestedLargeFloatArrayRead:  33099 ms       0.101        26.282       1087381
         ReflectNestedLargeFloatArrayWrite:  35159 ms       0.095        24.742       1087381
   ReflectNestedLargeFloatArrayBlockedRead:  33326 ms       0.100        26.302       1095674
  ReflectNestedLargeFloatArrayBlockedWrite:  36921 ms       0.090        23.741       1095674
{noformat}

After:
{noformat}
                                 test name     time    M entries/sec   M bytes/sec  bytes/cycle
                         ReflectRecordRead:   6058 ms       2.751       106.754        808498
                        ReflectRecordWrite:   3750 ms       4.444       172.470        808498
                      ReflectBigRecordRead:   6767 ms       1.478        90.709        767380
                     ReflectBigRecordWrite:   4433 ms       2.255       138.466        767380
                          ReflectFloatRead:   6155 ms       0.000       129.970       1000004
                         ReflectFloatWrite:   1083 ms       0.001       738.434       1000004
                         ReflectDoubleRead:   6610 ms       0.000       242.028       2000004
                        ReflectDoubleWrite:   1968 ms       0.000       812.864       2000004
                       ReflectIntArrayRead:   9462 ms       1.761        72.683        859709
                      ReflectIntArrayWrite:   2468 ms       6.751       278.584        859709
                      ReflectLongArrayRead:   5556 ms       1.500       115.941        805344
                     ReflectLongArrayWrite:   1851 ms       4.500       347.921        805344
                    ReflectDoubleArrayRead:   5755 ms       1.737       113.719        818144
                   ReflectDoubleArrayWrite:   1423 ms       7.027       459.900        818144
                     ReflectFloatArrayRead:  11373 ms       1.758        59.519        846172
                    ReflectFloatArrayWrite:   2267 ms       8.821       298.557        846172
               ReflectNestedFloatArrayRead:  12626 ms       1.584        53.610        846172
              ReflectNestedFloatArrayWrite:   4216 ms       4.743       160.542        846172
              ReflectNestedObjectArrayRead:  10522 ms       0.380        49.046        645104
             ReflectNestedObjectArrayWrite:   5565 ms       0.719        92.725        645104
          ReflectNestedLargeFloatArrayRead:   8022 ms       0.415       108.436       1087381
         ReflectNestedLargeFloatArrayWrite:   1788 ms       1.863       486.359       1087381
   ReflectNestedLargeFloatArrayBlockedRead:   8098 ms       0.412       108.799       1101357
  ReflectNestedLargeFloatArrayBlockedWrite:   7511 ms       0.444       117.295       1101357

{noformat}

                
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
> -------------------------------------------------------------------------------
>
>                 Key: AVRO-1282
>                 URL: https://issues.apache.org/jira/browse/AVRO-1282
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.4
>            Reporter: Leo Romanoff
>            Priority: Minor
>         Attachments: AVRO-1282-s1.patch, AVRO-1282-s2.patch, AVRO-1282-s3.patch, AVRO-1282-s5.patch,
AVRO-1282-s6.patch, avro-1282-v1.patch, avro-1282-v2.patch, avro-1282-v3.patch, avro-1282-v4.patch,
avro-1282-v5.patch, avro-1282-v6.patch, avro-1282-v7.patch, avro-1282-v8.patch, AVRO-1282-v9.patch,
TestUnsafeUtil.java
>
>
> Unsafe can be used to significantly speed up serialization process, if a JDK implementation
supports java.misc.Unsafe properly. Most JDKs running on PCs support it. Some platforms like
Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the reflection-based
approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick input/output.
>  
> 3) More over, Unsafe makes it possible to serialize to/deserialize from off-heap memory
directly and very quickly, without any intermediate buffers allocated on heap. There is virtually
no overhead compared to the usual byte arrays.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message