avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1175) BinaryData keeps a thread local reference after completing a compare, preventing compared arrays from being GC'd
Date Fri, 28 Sep 2012 19:07:07 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465831#comment-13465831
] 

Todd Lipcon commented on AVRO-1175:
-----------------------------------

Hmm, looking at the MR code, it appears like we usually call functions like JobConf.getOutputKeyComparator()
which make a new instance. So, if the comparator had some kind of deinit() function, we could
probably figure out where to call it. But making it part of the comparator (stateful) instead
of a threadlocal seems better all around.

We do have other examples of stateful comparators: WritableComparator itself uses a DataInputBuffer.
                
> BinaryData keeps a thread local reference after completing a compare, preventing compared
arrays from being GC'd
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: AVRO-1175
>                 URL: https://issues.apache.org/jira/browse/AVRO-1175
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.7.2
>            Reporter: Sandy Ryza
>            Assignee: Doug Cutting
>
> BinaryData holds on to BinaryDecoders as thread local variables (so it doesn't have to
make new ones for each compare?).  When a compare is completed, the BinaryDecoder still keeps
a reference to the ByteArrayByteSource, which stops its underlying byte array from being garbage
collected.
> This is causing an OutOfMemoryError in reducers when shuffling with MR2.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message