hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Mapreduce and Exclude
Date Fri, 15 Jan 2010 05:23:09 GMT
Hi David,

The ability to administratively blacklist TaskTrackers wasn't added until
0.21, which is not yet released.

In general, there isn't usually a great reason to do this -- you can take
down a tasktracker by just stopping it. The purpose of the HDFS-side
operation is to decommission a node gracefully while still allowing those
replicas to be served while they're being rereplicated elsewhere. For
MapReduce, this kind of graceful decommissioning is only useful if you (a)
can't ssh into the node for some reason and want to kick it out, or (b) have
some very long-running tasks that would be a pain to reschedule, so you want
to let those finish before shutting down a TT.

-Todd

On Thu, Jan 14, 2010 at 7:04 PM, David B. Ritch <david.ritch@gmail.com>wrote:

> We use dfs.exclude to point to a file containing a list of nodes with
> problems, and HDFS does not use those nodes.  We have mapred.exclude
> point to the same file, but the jobtracker still allows connections from
> those nodes.  Has anyone else gotten this to work?
>
> Further, it's easy to get the namenode to recognize changes in the file
> (hadoop dfsadmin -refreshNodes), but I do not see a way to get the
> jobtracker to re-read it.  Is there a way to do this?
>
> Thanks!
>
> David
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message