apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raja.Aravapalli <Raja.Aravapa...@target.com>
Subject Re: how to increase lifetime of hdfs delegation tokens ?
Date Tue, 05 Jul 2016 21:23:14 GMT

Hi Pramod,

As suggested in step2, I added the properties you provided below, and ran a new job, to see
the  debug lines "number of tokens: " and "updated token: "

And, as you said, I can see those lines in the application log now..

Does this mean, the renewal is working as expected ? So, will my application run continously
even after 7days now ? We have only changed refersh times here… how does this ensure the
application runs indefinite time…




Regards,
Raja.

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Saturday, July 2, 2016 at 9:52 AM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?


Thanks Pramod, I will these suggestions and let you know. Thanks a lot.


Regards,
Raja.

From: Pramod Immaneni <pramod@datatorrent.com<mailto:pramod@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Friday, July 1, 2016 at 8:36 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Hi Raja,

Some questions for you and also some options for you to verify without waiting for a long
time.

1. Do you see a warning message like ""No keytab specified for refreshing tokens, application
may not be able to run indefinitely" when application is being launched from command line.

2. For testing, set the parameters below in your dt-site.xml and launch your application.
This will make the application think that the tokens are only valid for 5 minutes and within
5 * 0.7 = 3.5 minutes (token refresh factor is 0.7) the application should try to get new
tokens. It should print the debug lines "number of tokens: " and "updated token: " in the
application master logs. The application master logs are in the log file of the first container
of the application. Let me know if you see those log lines.

<property>
        <name>dt.resourcemanager.delegation.token.max-lifetime</name>
        <value>300000</value>
</property>

<property>
        <name>dt.namenode.delegation.token.max-lifetime</name>
        <value>300000</value>
</property>

<property>
      <name>dt.attr.DEBUG</name>
      <value>true</value>
</property>

For more information about application auto-fetching new tokens read here

https://github.com/apache/apex-core/blob/master/docs/security.md

Thanks

On Fri, Jul 1, 2016 at 1:08 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Thanks a lot Pramod. Will wait for your response.


Regards,
Raja.

From: Pramod Immaneni <pramod@datatorrent.com<mailto:pramod@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Friday, July 1, 2016 at 10:56 AM

To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Hi Raja,

Let me look at this and get back to you.

Thanks

On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Can someone pls help me, how can I ensure, my apex application doesn’t fail after 7days…

Thanks a lot.


Regards,
Raja.

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Thursday, June 30, 2016 at 6:06 AM

To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?


Hi,

I triggered my application by specifying properties, “dt.authentication.principal” &
“dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”.

I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”.
But, still my apex application failed after 7days!!!

I am setting these properties in “properties.xml” file!

How can I ensure my settings are working correct. Having waiting for 7days to learn its failure
is a very tough thing. Hope there should be some other alternatives.

Can someone pls help me fix this ….  Thanks a lot !!


Regards,
Raja.

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, June 20, 2016 at 5:43 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?


Sure Pramod. Please respond on this mail chain when you get to know..

Thanks very much.


Regards,
Raja.

From: Pramod Immaneni <pramod@datatorrent.com<mailto:pramod@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Raja,

I believe it would. I will check and get back to you but the easiest way for you to check
is that the file should appear in HDFS under /user/<username>/datatorrent with the same
filename as it is in your local filesystem.

Thanks

On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Thanks for the response Pramod.

My quick question is, I see we should mention these properties in dt-site.xml !! I am not
sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass
some configuration to application.
Can I set these in properties.xml file and it will still work ?


Regards,
Raja.

From: Pramod Immaneni <pramod@datatorrent.com<mailto:pramod@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM

To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Hi Raja,

Yes the keytab would be copied over to HDFS and reused for getting a new token before the
old one expires. By default it is 7 days. If it is different in your cluster please set the
properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime
in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused
you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying
the property dt.authentication.store.keytab.All this is described in the document that Thomas
sent over.

Thanks

On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Hi Thomas,

To ensure auto renewal of delegation tokens life time, Can I use the the below properties
in properties.xml file ?


<property>
            <name>dt.authentication.principal</name>
            <value>kerberos-principal-of-user</value>
    </property>
    <property>
            <name>dt.authentication.keytab</name>
            <value>absolute-path-to-keytab-file</value>
    </property>

FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties
when launching apex applications in our secure hadoop environment, still they worked fine
without any issues, but failing after 7days!!

If I set the above properties in properties.xml, will that do auto-renewal and run successfully
without any issues of failing again due to delegation token lifetime expiry ??

Please advise.


Thanks a lot in advance.


Regards,
Raja.

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM

To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?


Thanks a lot Thomas.

Will take this as reference and test our application. Great!


Regards,
Raja.

From: Thomas Weise <thomas.weise@gmail.com<mailto:thomas.weise@gmail.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Token expiration working as expected!

Please have a look on how to extend or refresh it:

https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh

Thanks,
Thomas


On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Hi,

My Apex application failed exactly after running 7days in our distributed hadoop environment,
with delegation token expiry!!

Can someone pls help me with details, on how I can increase the delegation token time to lifetime
or any other process running in parallel to renew the tokens ?

Exception details below:


ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode
11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
        at org.apache.hadoop.ipc.Client.call(Client.java:1427)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)


Thanks a lot in advance.


Regards,
Raja.






Mime
View raw message