hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6606) Optimize encryption support in DataTransfer Protocol with High performance
Date Tue, 01 Jul 2014 08:20:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14048624#comment-14048624

Yi Liu commented on HDFS-6606:

Thanks [~tucu00], [~yoderme] and [~andrew.purtell@gmail.com] for your comments. 

I file JIRA HADOOP-10768 for Optimizing Hadoop RPC encryption performance. Not file that JIRA
before because 1) Hadoop utilizes SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for secure
authentication and data protection for RPC, not able to add custom encryption to them.  2)
PRC message is small, whether it is worth.
For #1, you remained me we could only use GssKrb5 to exchange user secrets, not do encryption
for whole RPC message, instead use the same way in this JIRA to encrypt RPC message. You are
For #2, we all think we can have benchmark to see real benefit, then we make a trade-off.

Thanks for the information, you are right, but it doesn't support AES-NI by default. Maybe
we can handle it in the same way as in this JIRA. It's more flexiable and can resolve encryption
issue of {{DIGEST-MD5}}. 

> Optimize encryption support in DataTransfer Protocol with High performance
> --------------------------------------------------------------------------
>                 Key: HDFS-6606
>                 URL: https://issues.apache.org/jira/browse/HDFS-6606
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, hdfs-client, security
>    Affects Versions: 3.0.0
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>             Fix For: 3.0.0
> In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol, it was a
great work.
> It utilizes SASL {{Digest-MD5}} mechanism,  it supports three security strength:
> * high                      3des   or rc4 (126bits)
> * medium             des or rc4(56bits)
> * low                       rc4(40bits)
> 3des and rc4 are slow, only *tens of MB/s*, 
> http://www.javamex.com/tutorials/cryptography/ciphers.shtml
> http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
> I will give more detailed performance data in future. Absolutely it’s bottleneck and
will vastly affect the end to end performance. 
> AES(Advanced Encryption Standard) is recommended as a replacement of DES, it’s more
secure; with AES-NI support, the throughput can reach nearly *2GB/s*, it won’t be the bottleneck
any more, AES and CryptoCodec work is supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693
(We may need to add a new mode support for AES). 
> This JIRA will use AES with AES-NI support as encryption algorithm for DataTransferProtocol.

This message was sent by Atlassian JIRA

View raw message