hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11885) createEncryptionZone should not block on initializing EDEK cache
Date Thu, 08 Jun 2017 18:39:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043213#comment-16043213

Xiao Chen commented on HDFS-11885:

Thanks Daryn, good question.

My understanding is mostly from HDFS-7209:
Currently when creating file in an encryption zone for the first time, key provider will get
bunch of keys from KMS and fill in the queue. It will take some time. We can initialize the
key queue when creating the encryption zone by admin.

Each create only gets 1 edek, and there's the async thread in {{ValueQueue}} to fill in the
cache (e.g. 500 edeks). I could see values by ensuring the cache to be filled proactively
rather than depending lazily on the first create. But maybe we can remove/reduce the sleep

Would also like to hear [~andrew.wang]'s ideas. :)

> createEncryptionZone should not block on initializing EDEK cache
> ----------------------------------------------------------------
>                 Key: HDFS-11885
>                 URL: https://issues.apache.org/jira/browse/HDFS-11885
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: encryption
>    Affects Versions: 2.6.5
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>            Priority: Critical
>         Attachments: HDFS-11885.001.patch, HDFS-11885.002.patch, HDFS-11885.003.patch
> When creating an encryption zone, we call {{ensureKeyIsInitialized}}, which calls {{provider.warmUpEncryptedKeys(keyName)}}.
This is a blocking call, which attempts to fill the key cache up to the low watermark.
> If the KMS is down or slow, this can take a very long time, and cause the createZone
RPC to fail with a timeout.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message