hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xianyin Xin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-5367) HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials
Date Wed, 13 Jul 2016 02:42:20 GMT
Xianyin Xin created YARN-5367:
---------------------------------

             Summary: HDFS delegation tokens in ApplicationSubmissionContext should be added
to systemCrednetials
                 Key: YARN-5367
                 URL: https://issues.apache.org/jira/browse/YARN-5367
             Project: Hadoop YARN
          Issue Type: Bug
          Components: yarn
            Reporter: Xianyin Xin
            Assignee: Xianyin Xin


App log aggregation may failed because of the below flow:
0) suppose the token.max-lifetime is 7 days and renew interval is 1 day;
1) start a long running job, like sparkJDBC, of which the AM acts as a service. When submitting
the job, HDFS token A in ApplicationSubmissionContext will be added to DelegationTokenRenewer,
but not added to systemCredentials;
2) after 1 day, submit a spark query. After received the query, AM will request containers
and start tasks. When start the containers, a new HDFS token B is used;
3) after 1 day, kill the job, when doing log aggregation, exception occurs which show token
B is not in the HDFS token cache so the connecting to HDFS fails;

We should add token A to systemCredentials to make sure token A can be delivered to NMs in
time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message