Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6EA02200CC8 for ; Fri, 14 Jul 2017 23:45:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6D2EE16E8F5; Fri, 14 Jul 2017 21:45:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B447916E8F1 for ; Fri, 14 Jul 2017 23:45:05 +0200 (CEST) Received: (qmail 68576 invoked by uid 500); 14 Jul 2017 21:45:04 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 68564 invoked by uid 99); 14 Jul 2017 21:45:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jul 2017 21:45:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 5CD6AC0EA0 for ; Fri, 14 Jul 2017 21:45:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ORk9QbvscS1I for ; Fri, 14 Jul 2017 21:45:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id C50265F5A2 for ; Fri, 14 Jul 2017 21:45:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id A1159E0E1C for ; Fri, 14 Jul 2017 21:45:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id DE11D24784 for ; Fri, 14 Jul 2017 21:45:00 +0000 (UTC) Date: Fri, 14 Jul 2017 21:45:00 +0000 (UTC) From: "Hudson (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-12140) Remove BPOfferService lock contention to get block pool id MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 14 Jul 2017 21:45:06 -0000 [ https://issues.apache.org/jira/browse/HDFS-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088144#comment-16088144 ] Hudson commented on HDFS-12140: ------------------------------- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12011 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12011/]) HDFS-12140. Remove BPOfferService lock contention to get block pool id. (kihwal: rev e7d187a1b6a826edd5bd0f708184d48f3674d489) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java > Remove BPOfferService lock contention to get block pool id > ---------------------------------------------------------- > > Key: HDFS-12140 > URL: https://issues.apache.org/jira/browse/HDFS-12140 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Affects Versions: 2.8.0 > Reporter: Daryn Sharp > Assignee: Daryn Sharp > Priority: Critical > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3 > > Attachments: HDFS-12140.branch-2.8.patch, HDFS-12140.trunk.patch > > > The block pool id is protected by a lock in {{BPOfferService}}. This creates excessive contention especially for xceivers threads attempting to queue IBRs and heartbeat processing. When the latter is delayed due to excessive FSDataset lock contention, it causes pipelines to collapse. > Accessing the block pool id should be lockless after registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org