hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10899) Add functionality to re-encrypt EDEKs
Date Fri, 18 Aug 2017 17:52:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133351#comment-16133351
] 

Xiao Chen commented on HDFS-10899:
----------------------------------

Just wanted to update about the myths with the 0.001 not 99% symptom. After much profiling
it turns out the contribution here is just CPU being stressed (while the thottler calculation
is based on lock time). Out of the lock there are various parts that's consuming CPU, from
the callable contacting KMS to the thread pool managing the futures.
If I change the ratio to 0.000001 (hence sleeping except for 1st batch), for my env the throughput
is at the same level of without re-encryption.

So I think this achieves our goal to allow a fast enough re-encryption by default, and leave
the configuration flexibility to slow re-encryption down to keep up throughput.

Closing up on a few TODOs around KMS error handling and fault injected unit testing, will
update soon.

> Add functionality to re-encrypt EDEKs
> -------------------------------------
>
>                 Key: HDFS-10899
>                 URL: https://issues.apache.org/jira/browse/HDFS-10899
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: encryption, kms
>            Reporter: Xiao Chen
>            Assignee: Xiao Chen
>         Attachments: editsStored, HDFS-10899.01.patch, HDFS-10899.02.patch, HDFS-10899.03.patch,
HDFS-10899.04.patch, HDFS-10899.05.patch, HDFS-10899.06.patch, HDFS-10899.07.patch, HDFS-10899.08.patch,
HDFS-10899.09.patch, HDFS-10899.10.patch, HDFS-10899.10.wip.patch, HDFS-10899.11.patch, HDFS-10899.12.patch,
HDFS-10899.13.patch, HDFS-10899.14.patch, HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt
edek design doc.pdf, Re-encrypt edek design doc V2.pdf
>
>
> Currently when an encryption zone (EZ) key is rotated, it only takes effect on new EDEKs.
We should provide a way to re-encrypt EDEKs after the EZ key rotation, for improved security.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message