apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raja.Aravapalli <Raja.Aravapa...@target.com>
Subject Re: how to increase lifetime of hdfs delegation tokens ?
Date Fri, 01 Jul 2016 06:20:06 GMT

Can someone pls help me, how can I ensure, my apex application doesn’t fail after 7days…

Thanks a lot.


Regards,
Raja.

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Thursday, June 30, 2016 at 6:06 AM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?


Hi,

I triggered my application by specifying properties, “dt.authentication.principal” &
“dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”.

I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”.
But, still my apex application failed after 7days!!!

I am setting these properties in “properties.xml” file!

How can I ensure my settings are working correct. Having waiting for 7days to learn its failure
is a very tough thing. Hope there should be some other alternatives.

Can someone pls help me fix this ….  Thanks a lot !!


Regards,
Raja.

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, June 20, 2016 at 5:43 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?


Sure Pramod. Please respond on this mail chain when you get to know..

Thanks very much.


Regards,
Raja.

From: Pramod Immaneni <pramod@datatorrent.com<mailto:pramod@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Raja,

I believe it would. I will check and get back to you but the easiest way for you to check
is that the file should appear in HDFS under /user/<username>/datatorrent with the same
filename as it is in your local filesystem.

Thanks

On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Thanks for the response Pramod.

My quick question is, I see we should mention these properties in dt-site.xml !! I am not
sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass
some configuration to application.
Can I set these in properties.xml file and it will still work ?


Regards,
Raja.

From: Pramod Immaneni <pramod@datatorrent.com<mailto:pramod@datatorrent.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM

To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Hi Raja,

Yes the keytab would be copied over to HDFS and reused for getting a new token before the
old one expires. By default it is 7 days. If it is different in your cluster please set the
properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime
in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused
you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying
the property dt.authentication.store.keytab.All this is described in the document that Thomas
sent over.

Thanks

On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Hi Thomas,

To ensure auto renewal of delegation tokens life time, Can I use the the below properties
in properties.xml file ?


<property>
            <name>dt.authentication.principal</name>
            <value>kerberos-principal-of-user</value>
    </property>
    <property>
            <name>dt.authentication.keytab</name>
            <value>absolute-path-to-keytab-file</value>
    </property>

FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties
when launching apex applications in our secure hadoop environment, still they worked fine
without any issues, but failing after 7days!!

If I set the above properties in properties.xml, will that do auto-renewal and run successfully
without any issues of failing again due to delegation token lifetime expiry ??

Please advise.


Thanks a lot in advance.


Regards,
Raja.

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM

To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?


Thanks a lot Thomas.

Will take this as reference and test our application. Great!


Regards,
Raja.

From: Thomas Weise <thomas.weise@gmail.com<mailto:thomas.weise@gmail.com>>
Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?

Token expiration working as expected!

Please have a look on how to extend or refresh it:

https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh

Thanks,
Thomas


On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Raja.Aravapalli@target.com<mailto:Raja.Aravapalli@target.com>>
wrote:

Hi,

My Apex application failed exactly after running 7days in our distributed hadoop environment,
with delegation token expiry!!

Can someone pls help me with details, on how I can increase the delegation token time to lifetime
or any other process running in parallel to renew the tokens ?

Exception details below:


ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode
11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
        at org.apache.hadoop.ipc.Client.call(Client.java:1427)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)


Thanks a lot in advance.


Regards,
Raja.




Mime
View raw message