From hdfs-issues-return-209462-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Tue Feb 6 04:17:11 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 51388180647 for ; Tue, 6 Feb 2018 04:17:11 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 40C97160C3B; Tue, 6 Feb 2018 03:17:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 89DD8160C5A for ; Tue, 6 Feb 2018 04:17:10 +0100 (CET) Received: (qmail 41631 invoked by uid 500); 6 Feb 2018 03:17:09 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 41620 invoked by uid 99); 6 Feb 2018 03:17:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Feb 2018 03:17:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 417631A090C for ; Tue, 6 Feb 2018 03:17:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -102.311 X-Spam-Level: X-Spam-Status: No, score=-102.311 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Gk-LTNvKylt3 for ; Tue, 6 Feb 2018 03:17:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id F1FE65FB93 for ; Tue, 6 Feb 2018 03:17:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 42FD7E008D for ; Tue, 6 Feb 2018 03:17:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 81B3F240E2 for ; Tue, 6 Feb 2018 03:17:00 +0000 (UTC) Date: Tue, 6 Feb 2018 03:17:00 +0000 (UTC) From: "Wangda Tan (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-12049) Recommissioning live nodes stalls the NN MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353288#comment-16353288 ] Wangda Tan commented on HDFS-12049: ----------------------------------- We plan to start merge vote of 3.1.0 on Feb 18, please let me know if any plan to finish this by Feb 18 or we need to move it to 3.2.0. > Recommissioning live nodes stalls the NN > ---------------------------------------- > > Key: HDFS-12049 > URL: https://issues.apache.org/jira/browse/HDFS-12049 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.6.0 > Reporter: Daryn Sharp > Priority: Critical > > A node refresh will recommission included nodes that are alive and in decommissioning or decommissioned state. The recommission will scan all blocks on the node, find over replicated blocks, chose an excess, queue an invalidate. > The process is expensive and worsened by overhead of storage types (even when not in use). It can be especially devastating because the write lock is held for the entire node refresh. _Recommissioning 67 nodes with ~500k blocks/node stalled rpc services for over 4 mins._ -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org