Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 438A0177C1 for ; Tue, 14 Oct 2014 05:43:44 +0000 (UTC) Received: (qmail 84122 invoked by uid 500); 14 Oct 2014 05:43:37 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 83999 invoked by uid 500); 14 Oct 2014 05:43:37 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 83987 invoked by uid 99); 14 Oct 2014 05:43:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Oct 2014 05:43:36 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rakeshr@huawei.com designates 119.145.14.65 as permitted sender) Received: from [119.145.14.65] (HELO szxga02-in.huawei.com) (119.145.14.65) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Oct 2014 05:43:09 +0000 Received: from 172.24.2.119 (EHLO szxeml419-hub.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id CAS36053; Tue, 14 Oct 2014 13:43:05 +0800 (CST) Received: from szxeml561-mbx.china.huawei.com ([169.254.5.20]) by szxeml419-hub.china.huawei.com ([10.82.67.158]) with mapi id 14.03.0158.001; Tue, 14 Oct 2014 13:42:56 +0800 From: Rakesh R To: "user@hadoop.apache.org" , "Dhiraj.Kamble@sandisk.com" Subject: RE: Redirect Writes/Reads to a particular Node Thread-Topic: Redirect Writes/Reads to a particular Node Thread-Index: Ac/nbNPJVdcp+vQUTP2lsCFBkNXjagAA7EXQ Date: Tue, 14 Oct 2014 05:42:56 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.170.130] Content-Type: multipart/alternative; boundary="_000_C2496325850AA74C92AAF83AA9662D26458DEF38szxeml561mbxchi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Virus-Checked: Checked by ClamAV on apache.org --_000_C2496325850AA74C92AAF83AA9662D26458DEF38szxeml561mbxchi_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Dhiraj, AFAIK there is a mechanism to pass the set of 'favoredNodes datanodes' that= should be favored as targets while creating a file. But this is only consi= dered as a hint, sometimes due to cluster state(for example: given dn doesn= 't have sufficient space, doesn't available etc.), namenode may not be able= to place the blocks on these datanodes then the hadoop replication works a= s per its policy. Please go through the below api which can be used for ach= ieving this behavior and hope this will help you to get some idea. DistributedFileSystem.java /* Same as * {@link #create(Path, FsPermission, boolean, int, short, long, * Progressable)} with the addition of favoredNodes that is a hint to * where the namenode should place the file blocks. * The favored nodes hint is not persisted in HDFS. Hence it may be honor= ed * at the creation time only. HDFS could move the blocks during balancing= or * replication, to move the blocks from favored nodes. A value of null me= ans * no favored nodes for this create */ public HdfsDataOutputStream create(final Path f, final FsPermission permission, final boolean overwrite, final int bufferSize, final short replication, final long blockSize, final Progressable progress, final InetSocketAddress[] favoredNodes) Regards, Rakesh From: Dhiraj Kamble [mailto:Dhiraj.Kamble@sandisk.com] Sent: 14 October 2014 10:38 To: user@hadoop.apache.org Subject: Redirect Writes/Reads to a particular Node Hi, Is it possible to redirect writes to one particular node i.e. store the pri= mary replica always on the same node; and have reads served from this prim= ary node. If the primary node goes down; then hadoop replication works as p= er its policy; but when this node comes up it should again become the prima= ry node. I don't see any config parameter available for core-site.xml or hd= fs-site.xml to serve this purpose. Is there any way I can do this. Regards, Dhiraj ________________________________ PLEASE NOTE: The information contained in this electronic mail message is i= ntended only for the use of the designated recipient(s) named above. If the= reader of this message is not the intended recipient, you are hereby notif= ied that you have received this message in error and that any review, disse= mination, distribution, or copying of this message is strictly prohibited. = If you have received this communication in error, please notify the sender = by telephone or e-mail (as shown above) immediately and destroy any and all= copies of this message in your possession (whether hard copies or electron= ically stored copies). --_000_C2496325850AA74C92AAF83AA9662D26458DEF38szxeml561mbxchi_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Dhiraj,

 

AFAIK there is a mecha= nism to pass the set of 'favoredNodes datanodes' that should be favored as = targets while creating a file. But this is only considered as a hint, somet= imes due to cluster state(for example: given dn doesn’t have sufficient space, doesn’t available etc.= ), namenode may not be able to place the blocks on these datanodes then the= hadoop replication works as per its policy. Please go through the below ap= i which can be used for achieving this behavior and hope this will help you to get some idea.

 

DistributedFileSyst= em.java

 

 /* Same as 

   * = {@link #create(Path, FsPermission, boolean, int, short, long,

   * Progressable)} with the addition of favoredNodes that is a hint to <= /span>

   * where the namenode should place the file blocks.

   * The favored nodes hint is not persisted in HDFS. Hence it may be hono= red

   * at the creatio= n time only. HDFS could move the blocks during balancing or

   * replication, t= o move the blocks from favored nodes. A value of null means

   * no favored nodes for this create

   */

  public HdfsData= OutputStream create(final Path f,

   &nbs= p;  final FsPermission permission, final boolean overwrite,=

   &nbs= p;  final int bufferSize, final short replication, final long blockSiz= e,

   &nbs= p;  final Progressable progress, final InetSocketAddress[] favoredNode= s)

 

Regards,

Rakesh

 

From: Dhiraj K= amble [mailto:Dhiraj.Kamble@sandisk.com]
Sent: 14 October 2014 10:38
To: user@hadoop.apache.org
Subject: Redirect Writes/Reads to a particular Node

 

Hi,

 

Is it possible to redirect writes to one particular = node i.e. store the primary replica  always on the same node; and have= reads served from this primary node. If the primary node goes down; then h= adoop replication works as per its policy; but when this node comes up it should again become the primary node. I don= ’t see any config parameter available for core-site.xml or hdfs-site.= xml to serve this purpose.

 

Is there any way I can do this.

 

 

Regards,

Dhiraj

 

 


PLEASE NOTE: The information contained in this electronic mail message is i= ntended only for the use of the designated recipient(s) named above. If the= reader of this message is not the intended recipient, you are hereby notif= ied that you have received this message in error and that any review, dissemination, distribution, or copy= ing of this message is strictly prohibited. If you have received this commu= nication in error, please notify the sender by telephone or e-mail (as show= n above) immediately and destroy any and all copies of this message in your possession (whether hard copies= or electronically stored copies).=

--_000_C2496325850AA74C92AAF83AA9662D26458DEF38szxeml561mbxchi_--