hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh R <rake...@huawei.com>
Subject RE: Redirect Writes/Reads to a particular Node
Date Tue, 14 Oct 2014 05:42:56 GMT
Hi Dhiraj,

AFAIK there is a mechanism to pass the set of 'favoredNodes datanodes' that should be favored
as targets while creating a file. But this is only considered as a hint, sometimes due to
cluster state(for example: given dn doesn't have sufficient space, doesn't available etc.),
namenode may not be able to place the blocks on these datanodes then the hadoop replication
works as per its policy. Please go through the below api which can be used for achieving this
behavior and hope this will help you to get some idea.


 /* Same as
   * {@link #create(Path, FsPermission, boolean, int, short, long,
   * Progressable)} with the addition of favoredNodes that is a hint to
   * where the namenode should place the file blocks.
   * The favored nodes hint is not persisted in HDFS. Hence it may be honored
   * at the creation time only. HDFS could move the blocks during balancing or
   * replication, to move the blocks from favored nodes. A value of null means
   * no favored nodes for this create
  public HdfsDataOutputStream create(final Path f,
      final FsPermission permission, final boolean overwrite,
      final int bufferSize, final short replication, final long blockSize,
      final Progressable progress, final InetSocketAddress[] favoredNodes)


From: Dhiraj Kamble [mailto:Dhiraj.Kamble@sandisk.com]
Sent: 14 October 2014 10:38
To: user@hadoop.apache.org
Subject: Redirect Writes/Reads to a particular Node


Is it possible to redirect writes to one particular node i.e. store the primary replica  always
on the same node; and have reads served from this primary node. If the primary node goes down;
then hadoop replication works as per its policy; but when this node comes up it should again
become the primary node. I don't see any config parameter available for core-site.xml or hdfs-site.xml
to serve this purpose.

Is there any way I can do this.



PLEASE NOTE: The information contained in this electronic mail message is intended only for
the use of the designated recipient(s) named above. If the reader of this message is not the
intended recipient, you are hereby notified that you have received this message in error and
that any review, dissemination, distribution, or copying of this message is strictly prohibited.
If you have received this communication in error, please notify the sender by telephone or
e-mail (as shown above) immediately and destroy any and all copies of this message in your
possession (whether hard copies or electronically stored copies).

View raw message