hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14705) Add batched reencryptEncryptedKey interface to KMS
Date Mon, 21 Aug 2017 20:51:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135797#comment-16135797
] 

Rushabh S Shah edited comment on HADOOP-14705 at 8/21/17 8:50 PM:
------------------------------------------------------------------

The last patch looks very very close.
{quote}
Not on the request itself, but the client sending it and the server receiving it both need
to be able to hold and parse it. As it turned out from HDFS-10899, bigger than 2k may trigger
edit log sync and impact performance.
For KMS here, I added a static 10k maxNumPerBatch as a safeguard too. Security-wise okay be
cause ACL is checked before iterating through the json payload.
{quote}
If I understand this comment correctly, for a batch of more than 2000, it will impact namenode
performance.
Then why do we have limit of {{5x}} on server side.
Am I missing something ?

Couple of minor comments.
+TestKeyProviderCryptoExtension.java+
Below line is from patch.
 bq. // Verify the decrypting the new EEK and orig EEK gives the same material. 
We should add the same check in {{TestKeyProviderCryptoExtension#testGenerateEncryptedKey}}
also.

+KMS.java+
Below line is from patch.
bq. LOG.trace("Exiting handleEncryptedKeyOp method.");
* Log line is carried over from {{handleEncryptedKeyOp}} method.
* If I read the existing method code properly, the log line {{Exiting <methodName>}}
is logged only when the call completes _gracefully_.
In case of any exceptions, all other calls logs the exception at {{debug}} level and don't
log the exiting log line.
Just to be consistent, we should also follow the same pattern in {{reencryptEncryptedKeys}}
method.


was (Author: shahrs87):
{quote}
Not on the request itself, but the client sending it and the server receiving it both need
to be able to hold and parse it. As it turned out from HDFS-10899, bigger than 2k may trigger
edit log sync and impact performance.
For KMS here, I added a static 10k maxNumPerBatch as a safeguard too. Security-wise okay be
cause ACL is checked before iterating through the json payload.
{quote}
If I understand this comment correctly, for a batch of more than 2000, it will impact namenode
performance.
Then why do we have limit of {{5x}} on server side.
Am I missing something ?

Couple of minor comments.
+TestKeyProviderCryptoExtension.java+
Below line is from patch.
 bq. // Verify the decrypting the new EEK and orig EEK gives the same material. 
We should add the same check in {{TestKeyProviderCryptoExtension#testGenerateEncryptedKey}}
also.

+KMS.java+
Below line is from patch.
bq. LOG.trace("Exiting handleEncryptedKeyOp method.");
* Log line is carried over from {{handleEncryptedKeyOp}} method.
* If I read the existing method code properly, the log line {{Exiting <methodName>}}
is logged only when the call completes _gracefully_.
In case of any exceptions, all other calls logs the exception at {{debug}} level and don't
log the exiting log line.
Just to be consistent, we should also follow the same pattern in {{reencryptEncryptedKeys}}
method.

> Add batched reencryptEncryptedKey interface to KMS
> --------------------------------------------------
>
>                 Key: HADOOP-14705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14705
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: kms
>            Reporter: Xiao Chen
>            Assignee: Xiao Chen
>         Attachments: HADOOP-14705.01.patch, HADOOP-14705.02.patch, HADOOP-14705.03.patch,
HADOOP-14705.04.patch, HADOOP-14705.05.patch, HADOOP-14705.06.patch, HADOOP-14705.07.patch,
HADOOP-14705.08.patch, HADOOP-14705.09.patch, HADOOP-14705.10.patch
>
>
> HADOOP-13827 already enabled the KMS to re-encrypt a {{EncryptedKeyVersion}}.
> As the performance results of HDFS-10899 turns out, communication overhead with the KMS
occupies the majority of the time. So this jira proposes to add a batched interface to re-encrypt
multiple EDEKs in 1 call.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message