hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission
Date Mon, 13 Jan 2014 23:27:56 GMT

    [ https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13870114#comment-13870114

Xuan Gong commented on YARN-1410:

Thanks, [~bikassaha] for the review.

bq. Xuan, can you please verify Karthik's comment above and fix/file jira?

replied. See my previous comment.

bq. Why is this at most once? This could be retried any number of times until the RM gives
us a new application id, right?

This AtMostOnce annotation is kind of confusing. we add annotation @AtMostOnce for "non-idempotent"
operations which need to do the retry if failover happens. So, I think that getNewApplication
should be retried if failover happens and it is "non-idempotent" operation. That is why I
add AtMostOnce annotation.

bq. In the common case, this extra operation is going to be pure overhead. Can we get this
information via an exception in the submitApplication() method?
bq. Why are we removing this code. We should return a new exception for the client.
bq. Can it happen that submitApplication() will fail on the client but the RM has actually
saved the application (with or without failover)? What should we do in that case?

Reply these three comments together. They are related.  Here are two scenarios:
1. If failover happens before YarnClient got the SubmitApplicationResponse, but state of RMApp
has been saved in zookeeper. When we retry the ClientRMService#submitApplication with the
old AppId, what will happen next is that we will get a "Application with id already exists"
 exception since we have already saved this application in zookeeper.
2. If we submit an application with the old ApplicationId. We expect that we will get a "Application
with id already exists" exception.

So, the main problem is how we can differentiate the two cases. If we get a "Application with
id already exists" exception, how can we that it is because of the failover or that we submit
an old applicationId ? Also, at ClientRMService#submitApplication, when the checker (rmContext.getRMApps().get(applicationId)
!= null) is true, how can we tell which exception we should throw ? Throw a "Application with
id already exists" exception because we submit the application with old id ? or throw some
other exceptions because of failover ?

So, that is why I move the applicationId duplicate checker from ClientRMService#submitApplication
into YarnClient#submitApplication. Before, we want to submit an application, make sure the
application id is not duplicate. After that, if we still have the checker (rmContext.getRMApps().get(applicationId)
!= null) as true in ClientRMService#submitApplication, we can assume that is because of failover.

bq. Can we create a common yarn client instead of doing it in every createApplication()/submitApplication()
helper method call?

Right. we can do that.

bq. What are the changes in TestYarnClient for? I dont see any new test added.

Those changes are related with corner case as I described.

bq. There is no testcase for the new functionality of YarnClient where it creates an appId
if its not supplied by the user? If you want, we could do it a separate jira related to this

Yes, I will add it.

> Handle client failover during 2 step client API's like app submission
> ---------------------------------------------------------------------
>                 Key: YARN-1410
>                 URL: https://issues.apache.org/jira/browse/YARN-1410
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Bikas Saha
>            Assignee: Xuan Gong
>         Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, YARN-1410.2.patch, YARN-1410.2.patch,
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> App submission involves
> 1) creating appId
> 2) using that appId to submit an ApplicationSubmissionContext to the user.
> The client may have obtained an appId from an RM, the RM may have failed over, and the
client may submit the app to the new RM.
> Since the new RM has a different notion of cluster timestamp (used to create app id)
the new RM may reject the app submission resulting in unexpected failure on the client side.
> The same may happen for other 2 step client API operations.

This message was sent by Atlassian JIRA

View raw message