From hdfs-issues-return-225204-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Mon Jul 2 12:02:09 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 63F1B180626 for ; Mon, 2 Jul 2018 12:02:08 +0200 (CEST) Received: (qmail 95275 invoked by uid 500); 2 Jul 2018 10:02:07 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 95260 invoked by uid 99); 2 Jul 2018 10:02:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2018 10:02:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id E92691A2A14 for ; Mon, 2 Jul 2018 10:02:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Tk1DAOOZV1GW for ; Mon, 2 Jul 2018 10:02:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id EFFE15F2C5 for ; Mon, 2 Jul 2018 10:02:05 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 82463E0DAA for ; Mon, 2 Jul 2018 10:02:05 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 3BD652183F for ; Mon, 2 Jul 2018 10:02:00 +0000 (UTC) Date: Mon, 2 Jul 2018 10:02:00 +0000 (UTC) From: "Andrew Wang (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529643#comment-16529643 ] Andrew Wang commented on HDFS-13703: ------------------------------------ +1 LGTM, will commit shortly. > Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit > ---------------------------------------------------------------------------- > > Key: HDFS-13703 > URL: https://issues.apache.org/jira/browse/HDFS-13703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance > Reporter: Todd Lipcon > Assignee: Todd Lipcon > Priority: Major > Attachments: hdfs-13703.patch, hdfs-13703.patch > > > The DFSClient creates a CorruptedBlocks object, which contains a HashMap, on every read call. In most cases, a read will not hit any corrupted blocks, and this hashmap is not used. It seems the JIT isn't smart enough to eliminate this allocation. We would be better off avoiding it and only allocating in the rare case when a corrupt block is hit. > Removing this allocation reduced CPU usage of a TeraValidate job by about 10%. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org