hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vihang Karajgaonkar (JIRA)" <>
Subject [jira] [Updated] (HIVE-17371) Move tokenstores to metastore module
Date Thu, 21 Sep 2017 16:05:00 GMT


Vihang Karajgaonkar updated HIVE-17371:
    Attachment: HIVE-17371.01.patch

Attaching the first version of the patch to trigger full suite of tests.

I think it makes more sense to move the tokenstore implementations to metastore instead of
having two copies of each in Hive and metastore. The TokenManager and SecretManager classes
can be kept as is to each consuming applications like HS2 or Metastore itself so that future
modifications can be done independently.

Here is the approach I followed.

1. Moved the {{HiveDelegationTokenManager}} and {{TokenStoreDelegationTokenSecretManager}}
from {{shims-common}} to {{service}} module since Metastore anyways has its own copy of TokenManager
and Metastore cannot have a dependency on shims-common.
2. Moved the {{MemoryTokenStore}}, {{ZooKeeperTokenStore}} and {{DBTokenStore}} to standalone-metastore
module. In order for these tokenstores to be consumable for multiple applications like HS2
and Metastore (and hopefully other applications in the future) we need to operate of higher
level abstraction of {{AbstractDelegationTokenIdentifier}} and {{DelegationTokenInformation}}.
3. Each application can define its DelegationTokenIndentifier like Hive already does in {{DelegationTokenIdentifier}}
(may be we should rename this to {{HiveDelegationTokenIdentifier}} to make it more explicit.
Similarly, metastore has its own implementation of  DelegationTokenIdentifier in {{DelegationTokenIdentifier}}
(again we should rename it possibly to say MetastoreDelegationTokenIndentifier)
4. Modified the shims-common classes of {{DelegationTokenSecretManager}} and {{DelegationTokenStore}}
to operate on hadoop level abstractions instead of Hive defined ones.

> Move tokenstores to metastore module
> ------------------------------------
>                 Key: HIVE-17371
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Metastore
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>         Attachments: HIVE-17371.01.patch
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and {{ZKTokenStore}}
since they implement {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of  {{}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws IOException {
>     String tokenStoreClassName =
>         MetastoreConf.getVar(conf, MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS,
>     // The second half of this if is to catch cases where users are passing in a HiveConf
>     // configuration.  It will have set the default value of
>     // " .class" to
>     // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its construction.
 But this is
>     // the hive-shims version of the memory store.  We want to convert this to our default
>     if (StringUtils.isBlank(tokenStoreClassName) ||
>         "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName))
>       return new MemoryTokenStore();
>     }
>     try {
>       Class<? extends DelegationTokenStore> storeClass =
>           Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
>       return ReflectionUtils.newInstance(storeClass, conf);
>     } catch (ClassNotFoundException e) {
>       throw new IOException("Error initializing delegation token store: " + tokenStoreClassName,
>     }
>   }
> {code}

This message was sent by Atlassian JIRA

View raw message