hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation
Date Fri, 27 Dec 2013 05:29:50 GMT

    [ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13857315#comment-13857315
] 

Xuan Gong commented on YARN-1521:
---------------------------------

an idempotent operation is one that has no additional effect if it is called more than once
with the same input parameters. That means no matter how many times you call the operation
with the same parameter, the result will be the same. In our HA case, i think that it means
no matter which RM is active, by calling the operations, we will get the same result. 

Based on this, here is my proposal:
{code}
In ApplicationClientProtocol : 
{code}
a. getNewApplication(), submitApplication(), forceKillApplication(), getApplicationReport(),
getApplications(),getQueueUserAclInfo(),getDelegationToken(),renewDelegationToken(), cancelDelegationToken()
need to add idempotent annotation.

When RM becomes Active, it will read back all the Applications, and all related DelegationTokens
back from the zk, in that case, no matter which RM is active, it will get all the information
for all the applications, either running or finished. That is the reason why we can always
get the same results if we call getApplicationReport(), getApplications(),getDelegationToken(),renewDelegationToken(),
cancelDelegationToken(). So, the idempotent annotation need to be added for those operations.

For forceKillApplication(), getNewApplication(), submitApplication(), those operation need
to add idempotent annotation too. Because, when the user calls them, no matter which RM is
active, the application will be submitted or killed.

For getQueueUserAclInfo(), YARN-1459 will guarantee that other RMs will get the latest configuration
if one of RMs are changed its configuration. In this case, when getQueueUserAclInfo is called,
we will always get the same results no matter which RM is active.


b. getClusterMetrics(), getClusterNodes(),getQueueInfo() do not need to add.

I do not want to add idempotent annotation. Because those are based on the Nodes information
or queue information. The current active RM has the resource information about all the NMs,
and queue info for all the applications. But when failover happens, the next active RMs will
not have all the NMs information immediately, and it does not have any queue info (such as
which application in which queue). 


{code}
In ResourceTracker:
{code}
a.registerNodeManager needs to add idempotent annotation.
Same reason why I want to add idempotent annotation for submitApplication()

b. nodeHeartbeat does not need to add

The new active RM does not know all the NMs immediately. 


{code}
In ApplicationMasterProtocol
{code}
Does not need to add for all the methods in ApplicationMasterProtocol.
There are three operations: registerApplicationMaster, finishApplicationMaster, allocate().
When the failover happens, a new RM will start a new Attempt for the running AM (just like
the RM restart). 


{code}
In ResourceManagerAdministrationProtocol
{code}
a.refreshQueues(), refreshNodes(), refreshSuperUserGroupsConfiguration(),RefreshUserToGroupsMappingsRequest(),
refreshAdminAcls(), refreshServiceAcls() need to add idempotent annotation.

YARN-1459 will guarantee that other RMs will get the latest configuration if one of RMs are
changed its configuration.

b.updateNodeResource() do not need to add
The new active RM does not know all the NMs immediately. 

> Mark appropriate protocol methods with the idempotent annotation
> ----------------------------------------------------------------
>
>                 Key: YARN-1521
>                 URL: https://issues.apache.org/jira/browse/YARN-1521
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is to identify
whether we need to add idempotent annotation and which methods can be marked as idempotent.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message