hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14333) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code
Date Thu, 20 Apr 2017 21:43:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977595#comment-15977595

Rushabh S Shah commented on HADOOP-14333:

If you read back [~daryn]'s [comment|https://issues.apache.org/jira/browse/HADOOP-14104?focusedCommentId=15924596&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15924596]
in HADOOP-14104, we discussed the exact same thing and came to decision that since DFSClient
is a private interface, we are allowed to change the signature of isHDFSEncryptionEnabled
and are allowed to throw IOException.
Since getTrashRoot was doing the wrong thing before HADOOP-14104 (Refer to https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2498),
we decided to do the same thing. At least that is limited to shell commands not jobs.

bq. Maybe we can temporarily change the behavior here to issue a warning message and return
false when exception is throw.
On this, I still don't think to return false in case of any exception.
If we want to temporarily fix hive use case, we can call {{DFSClient#getKeyProviderUri}} from
This means {{DFSClient#isHDFSEncryptionEnabled}} is just called from hive code.
Do something like this in {{DFSClient#isHDFSEncryptionEnabled}}
   * Probe for encryption enabled on this filesystem.
   * @return true if encryption is enabled
  public boolean isHDFSEncryptionEnabled() {
  try {
    URI keyProviderUri = getKeyProviderUri();
  } catch(IOException ioe) {
    return false;
  return keyProviderUri !=null
Probably add a comment on isHDFSEncryptionEnabled saying that this is here only to unblock
hive and no-one should use this.

> New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive
> ----------------------------------------------------------------------------------------------
>                 Key: HADOOP-14333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14333
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.8.1, 3.0.0-alpha3
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
> Though Hive should be fixed not to access DFSClient which is private to HADOOP, removing
the throws added by HADOOP-14104 is a quicker solution to unblock hive.
> Hive code
> {code}
> private boolean isEncryptionEnabled(DFSClient client, Configuration conf) {
>   try {
>     DFSClient.class.getMethod("isHDFSEncryptionEnabled");
>   } catch (NoSuchMethodException e) {
>     // the method is available since Hadoop-2.7.1
>     // if we run with an older Hadoop, check this ourselves
>     return !conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, "").isEmpty();
>   }
>   return client.isHDFSEncryptionEnabled();
> }
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message