hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10720) KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
Date Fri, 27 Jun 2014 05:42:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045560#comment-14045560

Alejandro Abdelnur commented on HADOOP-10720:

further details on how we could optimizing the extension operations in the KMS (server/client)
to reduce/eliminate calls from the NN on file create():

The {{KMSClientProvider}} would implement the {{CryptoExtensions}} interface, thus the {{KeyProviderExtensions}}
would delegate to it for generating and decrypting EDEKs.

The KMS client will have a queue with IV&EDEK per key. If the client queue is new/empty
a call to the KMS asking for N IV&EDEKs for the key will be send, the KMS response will
have at least one IV&EDEK. the returned IV&EDEKs are put int he queue and the first
entry from the queue is returned to the KMS client caller. The KMS client will have a daemon
thread that keeps requesting IV&EDEKs to the KMS for a key using a low and a high marks
to start and stop requesting more IV&EDEKs to the KMS. This means that most of the time
the KMS client will have IV&EDEKs locally cached for the NN to use.

On the KMS side a the same queue and replenisher daemon thread will have another cache of
IV&EDEK. On KMS client requests, typically, IV&EDEKS will not be generated synchronously
but simply drained from the queue. if the queue is new/empty a small number of IV&EDEKs
will be generated synchronously and returned to the caller. if the queue is not empty and
the client request N but the queue has less than N, the available elements will be returned
and the daemon thread will start producing more up to the high mark.

This dual cache will significantly reduce NN->KMS calls and reduce/eliminate the calls
that happen synchronously with NN create() call.
Suggestion: look at the JDK ThreadPoolExecutor uses a similar logic using wait/notify thus
not sleeping/looping-around.

> KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
> ---------------------------------------------------------------------------
>                 Key: HADOOP-10720
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10720
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 3.0.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Arun Suresh
>         Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch,
HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch
> KMS client/server should implement support for generating encrypted keys and decrypting
them via the REST API being introduced by HADOOP-10719.

This message was sent by Atlassian JIRA

View raw message