Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 25 Apr 2017 22:16:04 +0000 (UTC)
From: "Rushabh S Shah (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13066863.1493158201000.49246.1493158564319@Atlassian.JIRA>
In-Reply-To: <JIRA.13066863.1493158201000@Atlassian.JIRA>
References: <JIRA.13066863.1493158201000@Atlassian.JIRA> <JIRA.13066863.1493158201433@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HDFS-11702) Remove indefinite caching of key
 provider uri in DFSClient
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 25 Apr 2017 22:16:10 -0000


     [ https://issues.apache.org/jira/browse/HDFS-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rushabh S Shah updated HDFS-11702:
----------------------------------
    Description: 
There is an indefinite caching of key provider uri in dfsclient.
Relevant piece of code.
{code:title=DFSClient.java|borderStyle=solid}
  /**
   * The key provider uri is searched in the following order.
   * 1. If there is a mapping in Credential's secrets map for namenode uri.
   * 2. From namenode getServerDefaults rpc.
   * 3. Finally fallback to local conf.
   * @return keyProviderUri if found from either of above 3 cases,
   * null otherwise
   * @throws IOException
   */
  URI getKeyProviderUri() throws IOException {
    if (keyProviderUri != null) {
      return keyProviderUri;
    }
    // Lookup the secret in credentials object for namenodeuri.
    Credentials credentials = ugi.getCredentials();
   ...
   ...
{code}

Once the key provider uri is set, it won't refresh the value even if the key provider uri on namenode is changed.
For long running clients like on oozie servers, this means we have to bounce all the oozie servers to get the change reflected.
After this change, the client will cache the value for an hour after which it will issue getServerDefaults call and will refresh the key provider uri.

  was:
There is an indefinite caching of key provider uri in dfsclient.
Relevant piece of code.
{code:title=DFSClient.java|borderStyle=solid}
  /**
   * The key provider uri is searched in the following order.
   * 1. If there is a mapping in Credential's secrets map for namenode uri.
   * 2. From namenode getServerDefaults rpc.
   * 3. Finally fallback to local conf.
   * @return keyProviderUri if found from either of above 3 cases,
   * null otherwise
   * @throws IOException
   */
  URI getKeyProviderUri() throws IOException {
    if (keyProviderUri != null) {
      return keyProviderUri;
    }
{code}

Once the key provider uri is set, it won't refresh the value even if the key provider uri on namenode is changed.
For long running clients like on oozie servers, this means we have to bounce all the oozie servers to get the change reflected.
After this change, the client will cache the value for an hour after which it will issue getServerDefaults call and will refresh the key provider uri.


> Remove indefinite caching of key provider uri in DFSClient
> ----------------------------------------------------------
>
>                 Key: HDFS-11702
>                 URL: https://issues.apache.org/jira/browse/HDFS-11702
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>            Reporter: Rushabh S Shah
>            Assignee: Rushabh S Shah
>
> There is an indefinite caching of key provider uri in dfsclient.
> Relevant piece of code.
> {code:title=DFSClient.java|borderStyle=solid}
>   /**
>    * The key provider uri is searched in the following order.
>    * 1. If there is a mapping in Credential's secrets map for namenode uri.
>    * 2. From namenode getServerDefaults rpc.
>    * 3. Finally fallback to local conf.
>    * @return keyProviderUri if found from either of above 3 cases,
>    * null otherwise
>    * @throws IOException
>    */
>   URI getKeyProviderUri() throws IOException {
>     if (keyProviderUri != null) {
>       return keyProviderUri;
>     }
>     // Lookup the secret in credentials object for namenodeuri.
>     Credentials credentials = ugi.getCredentials();
>    ...
>    ...
> {code}
> Once the key provider uri is set, it won't refresh the value even if the key provider uri on namenode is changed.
> For long running clients like on oozie servers, this means we have to bounce all the oozie servers to get the change reflected.
> After this change, the client will cache the value for an hour after which it will issue getServerDefaults call and will refresh the key provider uri.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org