avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AVRO-743) Java: Performance Regression and memory pressure with GenericDatumReader
Date Wed, 09 Feb 2011 02:19:58 GMT

    [ https://issues.apache.org/jira/browse/AVRO-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992290#comment-12992290

Scott Carey commented on AVRO-743:

It should be as thread-safe as it was before the patch, so what worked before should continue
to work.   

In general, per thread is probably the way to go.  A static ThreadLocal<GenericDatumReader>
will work, or tie it to the consumer in the app in a more natural way (part of a thread pool?)
if there is one.

The fact is, the API is mutable, and if one thread calls setSchema() while another is reading
a datum, that contract is never going to be great.  Even fully synchronized it doesn't make
a lot of sense to do that.  

> Java: Performance Regression and memory pressure with GenericDatumReader
> ------------------------------------------------------------------------
>                 Key: AVRO-743
>                 URL: https://issues.apache.org/jira/browse/AVRO-743
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.5.0
>            Reporter: Scott Carey
>            Assignee: Scott Carey
>            Priority: Critical
>             Fix For: 1.5.0
>         Attachments: AVRO-743.patch
> AVRO-650 introduced a large performance regression and memory bloat issue with GenericDatumReader.
> Performance plummets for some Perf.java tests (One test took 1 hour to finish on my laptop).
> Some minor changes I tried result in it passing in shorter time, but with still an 80%
performance degredation.
> This is associated with memory bloat related to ThreadLocals.
> More details provided in comments.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message