hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <eri...@yahoo-inc.com>
Subject Re: Using Hadoop in a production environment
Date Wed, 14 Jun 2006 17:16:45 GMT
sounds pretty cool.

On Jun 14, 2006, at 9:23 AM, Runping Qi wrote:

>
> I have also been thinking of the Hadoop job scheduling issue too.
>
> In my applications, some jobs depend on the outputs of other jobs.
> Therefore, job dependency forms a DAG. A job is ready to run if and  
> only if
> it does not have any dependency or all the jobs it depends are  
> finished
> successfully. To help schedule and monitor a group of jobs like  
> that, I am
> thinking of implementing a utility class that:
> 	- accept jobs with dependency specification
>       - monitor job status
>       - submit jobs when they are ready
>
> With such a utility class, the application can construct its jobs,  
> specify
> their dependency and then hand the jobs to the utility class. The  
> utility
> class takes care of the details of job submission.
>
> Runping
>
>
>> -----Original Message-----
>> From: Paul Sutter [mailto:sutter@gmail.com]
>> Sent: Tuesday, June 13, 2006 5:51 PM
>> To: hadoop-dev@lucene.apache.org
>> Subject: Using Hadoop in a production environment
>>
>> We are starting to string together our disparate Hadoop jobs into a
>> running
>> system, and we have a couple issues that are coming up.
>>
>> I'm looking for feedback or suggestions on how we can solve them.
>>
>> (1) Scheduling Hadoop jobs
>>
>> Could an Ant extension be developed to let a complex set of Hadoop  
>> jobs be
>> controlled using a sort of a build script that decides which jobs  
>> need to
>> be
>> run?
>>
>> (2) How do we make Hadoop jobs atomic?
>>
>> One issue we have is that a failing job can leave directories in an
>> inconsistent format, making a mess for the other jobs.
>>
>> I'm thiking of an atomic operation we could submit to the  
>> nameserver. It
>> might consist of multiple directory deletions and directory  
>> renames, and
>> would either complete in entirety, or not at all. And in this way,  
>> we'd
>> get
>> the equivalent of a begin/commit/rollback capability for one simple
>> function.
>>
>> Im curious to hear others thoughts on this topic.
>


Mime
View raw message