hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-549) Asynchronous application submission
Date Sat, 06 Apr 2013 22:57:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624526#comment-13624526
] 

Bikas Saha commented on YARN-549:
---------------------------------

I dont think its an option for a client (eg YARNRunner) to not call getApplicationReport()
immediately after submitApplication() response comes back. At which point of time can it make
such a request definitively?

I am not clear about where the RMApp is being created. Currently ClientRMService.submitApplication()
calls RMAppManager.handle() which internally calls RMAppManager.submitApplication. Here, the
RMApp is created after checking for duplicates and some other initial book-keeping is done.
RM should not allow 2 submitApplication() requests to succeed if they have the same app id.
Hence, IMO, the initial app setup and addition to RMContext.apps must be done within the context
of the ClientRMService.submitApplication() thread. Perhaps by calling a synch RMAppManager.createApplication()
method which does this, followed by event sent to RMAppManager to start the app. OR ClientRMService.submitApplication()
itself adds new app to RMContext.apps and then sends event to RMAppManager. In either case,
the main slow piece of saving the app moves into RMApp transitions. This also enables clients
to immediately start querying the RM about the app state.

Clients dont need to change because even now an app is submitted only after it is accepted
by the scheduler. It would be helpful if YARNClientImpl has a helper function that does this
operation correctly so that library users dont have to bother about it. This is already suggested
in the doc.
                
> Asynchronous application submission
> -----------------------------------
>
>                 Key: YARN-549
>                 URL: https://issues.apache.org/jira/browse/YARN-549
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>         Attachments: Proposal of Asynchronous Application Submission_v1.pdf
>
>
> Currently, when submitting an application, storeApplication will be called for recovery.
However, it is a blocking API, and is likely to block concurrent application submissions.
Therefore, it is good to make application submission asynchronous, and postpone storeApplication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message