hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6592) Scheduler: Pause button desirable
Date Tue, 23 Feb 2010 07:13:28 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837117#action_12837117
] 

Arun C Murthy commented on HADOOP-6592:
---------------------------------------

This keeps coming up occasionally... 

Unfortunately people do not realize that this has significant (negative) consequences to the
cluster; in particular, map-outputs consume valuable temporary storage and make this feature
un-viable for Map-Reduce.

bq. If there's a bunch of "very low" priority tasks, they will still vie for resources. Similarly,
if I want to pause my task because I expect other tasks to arrive (i.e., there are none queuing),
very-low-pri doesn't quite fit the bill. 

Map-Reduce assumes that all dependencies are satisfied at time of job-submission...

bq. Similarly, if paused during the map phase, any reducers already allocated (which couldn't,
in theory, begin working until all mappers complete) would be killed and freed. (so yes, the
copy would need to restart, but the reducers wouldn't be held hostage by huge map tasks).

You need pre-emption, not pause.


> Scheduler: Pause button desirable
> ---------------------------------
>
>                 Key: HADOOP-6592
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6592
>             Project: Hadoop Common
>          Issue Type: Wish
>            Reporter: Adam Kramer
>            Priority: Minor
>
> It would be lovely if, from the jobtracker page, I could click a button that's not "kill"
or "fail" but ..."pause."
> The pause button would stop a certain task from starting any more mappers or reducers.
They would all wait in the "pending" stage until the job is "un-paused." Currently-running
tasks would continue to run, and then complete, thus freeing the resources for other jobs.
> This would help a lot for systems (esp. Hive) in which one or two jobs are hogging a
lot of mappers or reducers. The ones they have would finish, and then other jobs could "catch
up," and then they could be unpaused for a while. This would also allow for user-level throttling
of their jobs in instances where they need a lot of resources but have the time to spare.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message