hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-549) Asynchronous application submission
Date Mon, 15 Apr 2013 18:34:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631990#comment-13631990
] 

Zhijie Shen commented on YARN-549:
----------------------------------

After application store is moved out of ClientRMService#submitApplicaton, the remaining operations
in this function are:

1. Checking whether the submitted application is unique
2. Checking whether the requested AM resource is valid
3. Calling RMAppManager to handle the app submission event directly instead of scheduling
it through AsyncDispatcher

On the other side, RMAppManager#submitApplcation, which is ultimately called by ClientRMService#submitApplicaton
contain the following operations:

1. Checking whether the queue of ASC is null
2. Checking whether the application name is null
3. Creating the RMApp instance
4. Checking whether the submitted application is unique
5. Adding the RMApp instance to ApplicationACLsManager
6. Adding the RMApp instance to DelegationTokenRenewer
7. Kicking the RMApp instance to the state machine to start (or to be rejected if some exception
is caught)

IMHO, application submission can refactored with the following rules:
1. All the aforementioned operations are not slow, such that they don't need to be delayed.
2. It is confusing that part of sanity check is RMAppManager#submitApplcation while the other
is in ClientRMService#submitApplicaton. 1 and 2 of ClientRMService#submitApplicaton should
be moved into RMAppManager#submitApplcation. In addition, 4 of RMAppManager#submitApplcation
doesn't need to be done twice.
3. Instead of directly calling RMAppManager to handle the app submission event, ClientRMService#submitApplicaton
should call RMAppManager#submitApplcation directly, because the existing code is a fake event
scheduling method, and is somewhat misleading. Calling RMAppManager#submitApplcation directly
has the equivalent results. Then, in ClientRMService#submitApplicaton, there's some simple
code to call RMAppManager#submitApplcation and do the log.
4. RMAppManager#submitApplcation adds the "synchronized" modifier in order to prevent interleaving
the operations in RMAppManager#submitApplcation if two clients submitted applications simultaneously.
5. RMAppManager#submitApplcation throws YarnRemoteException, which will finally be passed
to the client if the submission has some exception, as what is done currently.

AFAIK, the aforementioned changes will not need the changes at the client side.
                
> Asynchronous application submission
> -----------------------------------
>
>                 Key: YARN-549
>                 URL: https://issues.apache.org/jira/browse/YARN-549
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>         Attachments: Proposal of Asynchronous Application Submission_v1.pdf
>
>
> Currently, when submitting an application, storeApplication will be called for recovery.
However, it is a blocking API, and is likely to block concurrent application submissions.
Therefore, it is good to make application submission asynchronous, and postpone storeApplication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message