hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi" <runp...@yahoo-inc.com>
Subject RE: Using Hadoop in a production environment
Date Wed, 14 Jun 2006 16:23:09 GMT

I have also been thinking of the Hadoop job scheduling issue too. 

In my applications, some jobs depend on the outputs of other jobs.
Therefore, job dependency forms a DAG. A job is ready to run if and only if
it does not have any dependency or all the jobs it depends are finished
successfully. To help schedule and monitor a group of jobs like that, I am
thinking of implementing a utility class that:
	- accept jobs with dependency specification
      - monitor job status
      - submit jobs when they are ready

With such a utility class, the application can construct its jobs, specify
their dependency and then hand the jobs to the utility class. The utility
class takes care of the details of job submission.


> -----Original Message-----
> From: Paul Sutter [mailto:sutter@gmail.com]
> Sent: Tuesday, June 13, 2006 5:51 PM
> To: hadoop-dev@lucene.apache.org
> Subject: Using Hadoop in a production environment
> We are starting to string together our disparate Hadoop jobs into a
> running
> system, and we have a couple issues that are coming up.
> I'm looking for feedback or suggestions on how we can solve them.
> (1) Scheduling Hadoop jobs
> Could an Ant extension be developed to let a complex set of Hadoop jobs be
> controlled using a sort of a build script that decides which jobs need to
> be
> run?
> (2) How do we make Hadoop jobs atomic?
> One issue we have is that a failing job can leave directories in an
> inconsistent format, making a mess for the other jobs.
> I'm thiking of an atomic operation we could submit to the nameserver. It
> might consist of multiple directory deletions and directory renames, and
> would either complete in entirety, or not at all. And in this way, we'd
> get
> the equivalent of a begin/commit/rollback capability for one simple
> function.
> Im curious to hear others thoughts on this topic.

View raw message