hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HADOOP-428) Condor and Hadoop Map Reduce integration
Date Wed, 23 May 2007 09:13:16 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Devaraj Das resolved HADOOP-428.

    Resolution: Won't Fix

Dropping this as HADOOP-1301 will take care of this feature.

> Condor and Hadoop Map Reduce integration
> ----------------------------------------
>                 Key: HADOOP-428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-428
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
> The issue is about using/enhancing Condor's features for Hadoop's Map Reduce framework.
Some of the early thoughts in this respect:
> * One should be able to submit a MR job that takes advantage of Condor's features like
node reservation according to a job's requirements, monitoring of jobs, etc.
> * JobTracker and TaskTrackers work as Master/Workers in the Condor environment. One should
be able to simply start a MR cluster and the cluster goes down when the job is done.
> * The classads can have an attribute for input file block locations that will be an input
to Condor's scheduling decisions.
> * Condor's features of monitoring jobs can be leveraged to reschedule failed TaskTrackers.
Checkpointing of JobTrackers can also probably be done so that if the JobTracker job dies
for some reason, the failed jobs can be restarted to start from the point where the JobTracker
was last checkpointed at (assuming the input data has not changed).
> * User priorities, job priorities should also be handled. If nodes are currently in use
due to a job being run by one user, and another user of the same priority submits a new job,
it gets queued and opportunistically the job of the second user is scheduled - for e.g., one
master and 1 worker to start with and then 2 workers and so on... If the second user is of
a higher priority, then the first user's job is completely suspended.
> Please add your thoughts on this topic.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message