hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3439) RM fails to renew token when Oozie launcher leaves before sub-job finishes
Date Thu, 02 Apr 2015 19:31:53 GMT

     [ https://issues.apache.org/jira/browse/YARN-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Lowe updated YARN-3439:
    Attachment: YARN-3439.001.patch

Daryn is out so posting a prototype patch he developed to get some early feedback.  Note that
this patch can't go in as-is, as it's a work-in-progress that hacks out the automatic HDFS
delegation token logic that was added as part of YARN-2704.

Essentially the idea is to reference count the tokens and only attempt to cancel them when
the token is no longer referenced.  Since the launcher job won't complete until it has successfully
submitted the sub-job(s), the token will remain referenced throughout the lifespan of the
workflow even if the launcher job exits early.

> RM fails to renew token when Oozie launcher leaves before sub-job finishes
> --------------------------------------------------------------------------
>                 Key: YARN-3439
>                 URL: https://issues.apache.org/jira/browse/YARN-3439
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Jason Lowe
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: YARN-3439.001.patch
> When the Oozie launcher runs a standard MapReduce job (not Pig) it doesn't linger waiting
for the sub-job to finish.  At that point the RM stops renewing delegation tokens for the
launcher job which wreaks havoc on the sub-job if the sub-job runs long enough for the tokens
to expire.

This message was sent by Atlassian JIRA

View raw message