avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Updated: (AVRO-743) Java: Performance Regression and memory pressure with GenericDatumReader
Date Sun, 06 Feb 2011 19:23:30 GMT

     [ https://issues.apache.org/jira/browse/AVRO-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Scott Carey updated AVRO-743:

    Status: Patch Available  (was: Open)

Patch implements a simple solution that worked best:

Keep track of the thread that created the GenericDatumReader, and allow only this thread to
access the local resolver reference.  Other threads use the global resolver cache.

Performance results:
                  GenericRead:  1824 ms,      3.654 million entries/sec.    141.829 million
           GenericNested_Read:  2940 ms,      2.267 million entries/sec.     87.989 million
      GenericWithDefault_Read:  3364 ms,      1.981 million entries/sec.     76.898 million
   GenericWithOutOfOrder_Read:  1799 ms,      3.705 million entries/sec.    143.792 million
    GenericWithPromotion_Read:  2020 ms,      3.300 million entries/sec.    128.059 million
GenericOneTimeDecoderUse_Read:  1813 ms,      3.676 million entries/sec.    142.682 million
 GenericOneTimeReaderUse_Read:  2290 ms,      2.910 million entries/sec.    112.944 million
       GenericOneTimeUse_Read:  2253 ms,      2.959 million entries/sec.    114.831 million

It is just as thread-safe as the prior version.  If multiple threads are concurrently doing
setSchema() or setExcpeted() other threads may not see the changes at the same time.  It should
be stable, and eventually consistent.  If AVRO-650 was good enough, this should be too.  

> Java: Performance Regression and memory pressure with GenericDatumReader
> ------------------------------------------------------------------------
>                 Key: AVRO-743
>                 URL: https://issues.apache.org/jira/browse/AVRO-743
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.5.0
>            Reporter: Scott Carey
>            Assignee: Scott Carey
>            Priority: Critical
>             Fix For: 1.5.0
>         Attachments: AVRO-743.patch
> AVRO-650 introduced a large performance regression and memory bloat issue with GenericDatumReader.
> Performance plummets for some Perf.java tests (One test took 1 hour to finish on my laptop).
> Some minor changes I tried result in it passing in shorter time, but with still an 80%
performance degredation.
> This is associated with memory bloat related to ThreadLocals.
> More details provided in comments.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message