hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shevek (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2491) generalize the TT / JT servers to handle more generic tasks
Date Wed, 22 Apr 2009 14:54:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701557#action_12701557

Shevek commented on HADOOP-2491:

I appreciate that I am a new contributor to Hadoop, but I am very interested in running MR
tasks on Hadoop which are not as strictly governed by the current MR workflow as is the current
implementation. I am currently reading the source code to Hadoop, I'm slightly stuck in the
gap between what appears to be a version 2.0 implementation emerging from the works, but even
so, I am very interested in contributing towards the effort suggested by this ticket. I have
spent the last couple of years writing high-level compilers for MR-like languages, and hope
this expertise can be of use.

> generalize the TT / JT servers to handle more generic tasks
> -----------------------------------------------------------
>                 Key: HADOOP-2491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2491
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: eric baldeschwieler
> We've been discussing a proposal to generalize the TT / JT servers to handle more generic
tasks and move job specific work out of the job tracker and into client code so the whole
system is both much more general and has more coherent layering.  The result would look more
like condor/pbs like systems (or presumably borg) with map-reduce as a user job.
> Such a system would allow the current map-reduce code to coexist with other work-queuing
libraries or maybe even persistent services on the same Hadoop cluster, although that would
be a stretch goal.  We'll kick off a thread with some documents soon.
> Our primary goal in going this way would be to get better utilization out of map-reduce
clusters and support a richer scheduling model.  The ability to support alternative job frameworks
would just be gravy!
> ----
> Putting this in as a place holder.  Hope to get folks talking about this to post some
more detail.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message