hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit
Date Mon, 02 Jul 2018 10:02:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529643#comment-16529643
] 

Andrew Wang commented on HDFS-13703:
------------------------------------

+1 LGTM, will commit shortly.

> Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-13703
>                 URL: https://issues.apache.org/jira/browse/HDFS-13703
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: performance
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Major
>         Attachments: hdfs-13703.patch, hdfs-13703.patch
>
>
> The DFSClient creates a CorruptedBlocks object, which contains a HashMap, on every read
call. In most cases, a read will not hit any corrupted blocks, and this hashmap is not used.
It seems the JIT isn't smart enough to eliminate this allocation. We would be better off avoiding
it and only allocating in the rare case when a corrupt block is hit.
> Removing this allocation reduced CPU usage of a TeraValidate job by about 10%.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message