hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5611) Provide an API to update lifetime of an application.
Date Wed, 19 Oct 2016 09:39:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588202#comment-15588202

Rohith Sharma K S commented on YARN-5611:

Summary of discussion for supporting  update API for timeout with [~vinodkv], [~vvasudev]
and [~sunilg].

*Current Patch Approach*: update API is synchronous call. And its timeout parameter affect
from now i.e targetTimeoutFromNow(in seconds) for given ApplicationTimeoutTypes. Say, user
wants to update timeout, if user passes 30 seconds for update then RM will calculate (now
+ 30 seconds ) and store in state store. 

*Concern on update API approach* 
# synchronous call is too expensive. In case of statestore problem, other RPC calls are blocked.
So basically approach of synchronous call need to be changed to asynchronous call. Note, updatePriority
is also blocking call which need to be solved once server side supported. 

# Yarn-client need to be adopt polling mechanism for RM for its update confirmation. The first
important thing need consider for polling mechanism is *Idempotent* API.  So, to the user,
*input remain relative*, but client need to convert relative time to targetedTime and send
a request to RM with targeted-time. The RM takes requested targeted timeout as-is and use
it. Say, user input timeout as 30 seconds, then yarn client change it to targetedTimeout=(now
+ 30seconds) and sends to RM. To the RM, update request is always targeted time.
# Multiple client update timeout for same application. Here, issue is how does yarn-client
get to know about its update confirmation.? How long yarn-client need to re-try/wait before
bailing out.? 
# *More importantly, Timezone issue*  if yarnclient converts it as explained in first challenge.
 Say, RM is running in IST , and yarnclient is running in PDT timezone. Even though Yarnclient
converts user input to targeted time and send to RM, since both client-server timezone are
different, the targetedTimeout value is totally different.

*Some of the thoughts solving the challenges*
# Server(RM)
## Always accept targeted timeout in UTC format and store in state-store as UTC format only.
For RM monitoring, covert UTC timestamp to local machine timezone and use it. This solves
Timezone issue from RM.
## Once update request received by RM, firstly RM should store new value in statestore. Secondly,
update to monitoring service. 
## RM should check targeted timeout is same as request time.If yes, return a boolean flag
indicating update is success. Always keep a copy of updated timeout values in RMApp and return
these values when somebody query for timeout values. 
# YarnClient:
## Convert user input timeout to targeted-timeout in UTC and send update request to RM
## Keep sending update request in loop as long as response received as true indicating update
## Bail out and fail the client operation after some configured value. 

Any comments on approach are welcome. 

> Provide an API to update lifetime of an application.
> ----------------------------------------------------
>                 Key: YARN-5611
>                 URL: https://issues.apache.org/jira/browse/YARN-5611
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>         Attachments: 0001-YARN-5611.patch, 0002-YARN-5611.patch, 0003-YARN-5611.patch,
> YARN-4205 monitors an Lifetime of an applications is monitored if required. 
> Add an client api to update lifetime of an application. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message