hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj K (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3813) Support Application timeout feature in YARN.
Date Thu, 09 Jul 2015 02:25:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619755#comment-14619755
] 

Devaraj K commented on YARN-3813:
---------------------------------

Thanks [~nijel] and [~rohithsharma] for the design proposal.

{quote}
New auxillary service : RMAppTimeOutService
Responsibility is to track the running application. Simple logic

//if job is running and the time elapsed kill
if ((RMAppState == SUBMITTED/ACCEPTED/RUNNING) &&
&& (currentTime - app.getSubmitTime()) >= timeout
{quote}

How frequently are you going to check this condition for each application?

Can we have a monitor something like RMAppTimeOutMonitor which extends AbstractLivelinessMonitor,
when the application gets submitted to RM then we can register the application with RMAppTimeOutMonitor
using the user specified timeout. And when the timeout reaches, RMAppTimeOutMonitor can trigger
an event to take an action further.

bq. Yes, having a separate TIMEOUT event and TIMEOUT state is good approach and other option.
Initially we consider to have new state TIMEOUT which require very huge changes across all
the modules.
I feel having a TIMEOUT state for RMAppImpl  would be proper here. When RMAppTimeOutMonitor
triggers an event on timeout for an application, RMAppImpl can move the state to TIMEOUT state
from any of the non-final states and during the transition it can handle stopping the running
attempt and the containers. I don't see here that there will be so many changes required for
achieving it.


> Support Application timeout feature in YARN. 
> ---------------------------------------------
>
>                 Key: YARN-3813
>                 URL: https://issues.apache.org/jira/browse/YARN-3813
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: nijel
>         Attachments: YARN Application Timeout .pdf
>
>
> It will be useful to support Application Timeout in YARN. Some use cases are not worried
about the output of the applications if the application is not completed in a specific time.

> *Background:*
> The requirement is to show the CDR statistics of last few  minutes, say for every 5 minutes.
The same Job will run continuously with different dataset.
> So one job will be started in every 5 minutes. The estimate time for this task is 2 minutes
or lesser time. 
> If the application is not completing in the given time the output is not useful.
> *Proposal*
> So idea is to support application timeout, with which timeout parameter is given while
submitting the job. 
> Here, user is expecting to finish (complete or kill) the application in the given time.
> One option for us is to move this logic to Application client (who submit the job). 
> But it will be nice if it can be generic logic and can make more robust.
> Kindly provide your suggestions/opinion on this feature. If it sounds good, i will update
the design doc and prototype patch



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message