hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10587) Use a thread-local cache in TokenIdentifier#getBytes to avoid creating many DataOutputBuffer objects
Date Sat, 10 May 2014 22:13:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993079#comment-13993079
] 

Colin Patrick McCabe commented on HADOOP-10587:
-----------------------------------------------

Recently, I encountered an edit log with many {{DELEGATION_TOKEN_IDENTIFIER}} log entries.
 Attempts to load this edit log failed with an out-of-memory error, occurring in {{TokenIdentifier#getBytes}}.
 While the edit log seemed to have an unusally large number of these log entries due to a
misconfiguration, I think it would also be useful to optimize {{TokenIdentifier#getBytes}}
to reduce this memory consumption somewhat.

> Use a thread-local cache in TokenIdentifier#getBytes to avoid creating many DataOutputBuffer
objects
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-10587
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10587
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 2.4.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HADOOP-10587.001.patch
>
>
> We can use a thread-local cache in TokenIdentifier#getBytes to avoid creating many DataOutputBuffer
objects.  This will reduce our memory usage (for example, when loading edit logs), and help
prevent OOMs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message