hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3740) Make JobInProgress pluggable
Date Mon, 14 Jul 2008 12:21:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613304#action_12613304

Amar Kamat commented on HADOOP-3740:

bq. allow users to override part of it dynamically
I dont think allowing user code to run in JT is a good idea. I think it makes more sense to
keep the scheduling code as is and spawn tasks irrespective of the resource. The M/R code
should (as a first step) try and get the license (e.g in {{configure()}}) and block on the
{{resource.get()}} call. It might seem like wastage of resources but probably better than
running user code in Hadoop daemons.  Also license is one type of resource. There can be different
types of resources and hence having separate hooks might not make sense. Having a protocol
that defines a communication between the scheduler (as of today the JIP) and implemented by
the _resource-server_ might be another way to tackle this problem.

> Make JobInProgress pluggable
> ----------------------------
>                 Key: HADOOP-3740
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3740
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
> By allowing a pluggable JobInProgess it will be possible for provide implementations
that can do a sophisticated task provisioning to the JobTracker. 
> For example, by providing alternate implementations of the {{obtainNewMapTask}}, {{obtainNewReduceTask}}
and {{updateTaskInProgress}} it would be possible to implement a license server that allows
to throttle use of external resources (ie webservices, databases) so at any given time there
are not more than N tasks using a given resource. For this a task could be tagged with the
names of external resources and the license server would keep track of the tasks running per
tag, if the counter reaches zero then the {{obtainNew*Task}} method could return NULL instead
of a task.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message