hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ioan Zeng <zengi...@gmail.com>
Subject YARN Features
Date Tue, 12 Mar 2013 14:31:55 GMT
Hello Hadoop Team,

I need some help/support in evaluating Hadoop YARN from the perspective of:

- Adding/removing test nodes dynamically without restart
 Is this possible in YARN? or is it only a MapReduce application feature?

- Failover from crashed test nodes
 Is this automatically done by the YARN framework?

- Simple prioritization of jobs
 I know there is a prioritization possibility of the tasks in the
Scheduler, right?

- Insight into queue (simple reporting, including running jobs on nodes)
 Is there any reporting possibility in YARN regarding the status of
the not executed, running, finished jobs?

- Support heterogenous OSes on nodes (or use separate masters for
homogenous grids)
 I think this is supported, right?

Another point I would like to evaluate is the Distributed Shell example usage.
Our use case is to start different scripts on a grid. Once a node has
finished a script a new script has to be started on it. A report about
the scripts execution has to be provided. in case a node has failed to
execute a script it should be re-executed on a different node. Some
scripts are Windows specific other are Unix specific and have to be
executed on a node with a specific OS.

The question is:
Would it be feasible to adapt the example "Distributed Shell"
application to have the above features?
If yes how could I run some specific scripts only on a specific OS? Is
this the ResourceManager responsability? What happens if there is no
Windows node for example in the grid but in the queue there is a
Windows script?
How to re-execute failed scripts? Does it have to be implemented by
custom code, or is it a built in feature of YARN?

Thank you in advance for your support,
Ioan Zeng

View raw message