hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10899) Add functionality to re-encrypt EDEKs
Date Tue, 22 Aug 2017 00:32:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136093#comment-16136093

Wei-Chiu Chuang commented on HDFS-10899:


private void throttle() throws InterruptedException {
  if (throttleLimitRatio >= 1.0)
the default is 1.0, which means no throttle for updater, and updater would keep contending
for namenode lock even if it has nothing to do?

LOG.warn("Failed to update re-encrypted edek to xattr for file {}",
 zonePath is not a file name. 

Please log the ioe as well.

LinkedList<Future> tasks = zst.getTasks();
ReencryptionHandler and ReencryptionUpdater shares ZoneSubmissionTracker#tasks.

Access to zst.getTasks() is protected in processCheckpoints() by a FSDirectory write lock
and a FSNamesystem write lock, whereas it is protected in ReencryptionHandler#submitCurrentBatch
by a FSDirectory read lock and a FSNameSystem readlock. If there will always be just one ReencryptionHandler,
then this is okay.

List<XAttr> xAttrs = null;

while( … ) {
  xAttrs = Lists.newArrayListWithCapacity(1);

  final XAttr xattr = FSDirEncryptionZoneOp
    .updateReencryptionProgress(dir, zoneNode,
status, task.lastFile,
        task.numFilesUpdated, task.numFailures);
return xAttrs;
Looks like you allocate a new xAttrs array for each task, but then discard it, only to return
the the xAttrs array for the last task?
I guess you want to instantiate the xAttrs array outside of the while loop?

the edit log is written only when all tasks are successful. If some tasks fail, the successful
tasks would record xattrs in memory, but will not get written into edit log.

> Add functionality to re-encrypt EDEKs
> -------------------------------------
>                 Key: HDFS-10899
>                 URL: https://issues.apache.org/jira/browse/HDFS-10899
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: encryption, kms
>            Reporter: Xiao Chen
>            Assignee: Xiao Chen
>         Attachments: editsStored, HDFS-10899.01.patch, HDFS-10899.02.patch, HDFS-10899.03.patch,
HDFS-10899.04.patch, HDFS-10899.05.patch, HDFS-10899.06.patch, HDFS-10899.07.patch, HDFS-10899.08.patch,
HDFS-10899.09.patch, HDFS-10899.10.patch, HDFS-10899.10.wip.patch, HDFS-10899.11.patch, HDFS-10899.12.patch,
HDFS-10899.13.patch, HDFS-10899.14.patch, HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt
edek design doc.pdf, Re-encrypt edek design doc V2.pdf
> Currently when an encryption zone (EZ) key is rotated, it only takes effect on new EDEKs.
We should provide a way to re-encrypt EDEKs after the EZ key rotation, for improved security.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message