hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eric baldeschwieler (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2491) generalize the TT / JT servers to handle more generic tasks
Date Wed, 26 Dec 2007 21:23:42 GMT
generalize the TT / JT servers to handle more generic tasks
-----------------------------------------------------------

                 Key: HADOOP-2491
                 URL: https://issues.apache.org/jira/browse/HADOOP-2491
             Project: Hadoop
          Issue Type: Improvement
          Components: mapred
            Reporter: eric baldeschwieler


We've been discussing a proposal to generalize the TT / JT servers to handle more generic
tasks and move job specific work out of the job tracker and into client code so the whole
system is both much more general and has more coherent layering.  The result would look more
like condor/pbs like systems (or presumably borg) with map-reduce as a user job.

Such a system would allow the current map-reduce code to coexist with other work-queuing libraries
or maybe even persistent services on the same Hadoop cluster, although that would be a stretch
goal.  We'll kick off a thread with some documents soon.

Our primary goal in going this way would be to get better utilization out of map-reduce clusters
and support a richer scheduling model.  The ability to support alternative job frameworks
would just be gravy!

----

Putting this in as a place holder.  Hope to get folks talking about this to post some more
detail.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message