From hdfs-issues-return-216619-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Apr 13 20:33:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 81E28180627 for ; Fri, 13 Apr 2018 20:33:04 +0200 (CEST) Received: (qmail 32160 invoked by uid 500); 13 Apr 2018 18:33:03 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 32149 invoked by uid 99); 13 Apr 2018 18:33:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Apr 2018 18:33:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C6340C2BFD for ; Fri, 13 Apr 2018 18:33:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 5LcjTP9BEzuf for ; Fri, 13 Apr 2018 18:33:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 611EA5FAEB for ; Fri, 13 Apr 2018 18:33:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7A902E00A7 for ; Fri, 13 Apr 2018 18:33:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 33E6A2120A for ; Fri, 13 Apr 2018 18:33:00 +0000 (UTC) Date: Fri, 13 Apr 2018 18:33:00 +0000 (UTC) From: "BELUGA BEHR (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HDFS-13448) HDFS Block Placement - Ignore Locality MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HDFS-13448: ------------------------------- Status: Patch Available (was: Open) > HDFS Block Placement - Ignore Locality > -------------------------------------- > > Key: HDFS-13448 > URL: https://issues.apache.org/jira/browse/HDFS-13448 > Project: Hadoop HDFS > Issue Type: New Feature > Components: block placement, hdfs-client > Affects Versions: 3.0.1, 2.9.0 > Reporter: BELUGA BEHR > Assignee: BELUGA BEHR > Priority: Minor > Attachments: HDFS-13448.1.patch > > > According to the HDFS Block Place Rules: > {quote} > /** > * The replica placement strategy is that if the writer is on a datanode, > * the 1st replica is placed on the local machine, > * otherwise a random datanode. The 2nd replica is placed on a datanode > * that is on a different rack. The 3rd replica is placed on a datanode > * which is on a different node of the rack as the second replica. > */ > {quote} > However, there is a hint for the hdfs-client that allows the block placement request to not put a block replica on the local datanode _where 'local' means the same host as the client is being run on._ > {quote} > /** > * Advise that a block replica NOT be written to the local DataNode where > * 'local' means the same host as the client is being run on. > * > * @see CreateFlag#NO_LOCAL_WRITE > */ > {quote} > I propose that we add a new flag that allows the hdfs-client to request that the first block replica be placed on a random DataNode in the cluster. The subsequent block replicas should follow the normal block placement rules. > The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block replica is not placed on the local node, but it is still placed on the local rack. Where this comes into play is where you have, for example, a flume agent that is loading data into HDFS. > If the Flume agent is running on a DataNode, then by default, the DataNode local to the Flume agent will always get the first block replica and this leads to un-even block placements, with the local node always filling up faster than any other node in the cluster. > Modifying this example, if the DataNode is removed from the host where the Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then the default block placement policy will still prefer the local rack. This remedies the situation only so far as now the first block replica will always be distributed to a DataNode on the local rack. > This new flag would allow a single Flume agent to distribute the blocks randomly, evenly, over the entire cluster instead of hot-spotting the local node or the local rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org