hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
Date Mon, 04 Jun 2018 18:21:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500669#comment-16500669

Daryn Sharp commented on HADOOP-10768:

{quote}Yes, shouldAuthenticateOverKrb() would throw NPE in some cases, we should catch it.
No, catching and ignoring all exceptions from shouldAuthenticateOverKrb is completely wrong.

I appreciate all the work here.  However I have a number of security concerns.  I didn't
care too much before other than not breaking the protocol and not making simple mistakes.
 Times change.  I care a lot now.

I don't fully understand what's going on in the key negotiation, don't feel qualified to determine
if it's secure (although it doesn't appear to be), doubt few in our community are either,
now and in the future.  I don't like being tied to AES/CTR+custom-hmac when other ciphers
like AES/GCM/ECDHE may be a requirement.  

An actual SSL implementation written by experts that we don't maintain and is tuned for performance
is a far preferable approach.  To that end, I've made surprising small tweaks to the ipc server
to use netty (configurable).  Goal is dropping in netty's SSL handlers.

Note that hadoop's ipc is pretty darn optimized.  I was surprised to find netty initially
incurred a 20% performance hit.  I've since closed the gap to <5%.  Note that one of
the challenges was avoiding array copies and allocations.  I see so much in the current patch
that it's likely what's sapping performance so much.

I'm finishing up debugging a few failed test cases.  Should post on a different Jira this
afternoon.  Then we can work on the ipc client.







> Optimize Hadoop RPC encryption performance
> ------------------------------------------
>                 Key: HADOOP-10768
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10768
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: performance, security
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Yi Liu
>            Assignee: Dapeng Sun
>            Priority: Major
>         Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch, HADOOP-10768.003.patch,
HADOOP-10768.004.patch, HADOOP-10768.005.patch, HADOOP-10768.006.patch, HADOOP-10768.007.patch,
HADOOP-10768.008.patch, HADOOP-10768.009.patch, HADOOP-10768.010.patch, HADOOP-10768.011.patch,
Optimize Hadoop RPC encryption performance.pdf, cpu_profile_RPC_encryption_AES.png, cpu_profile_rpc_encryption_optimize_calculateHMAC.png
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to "privacy". It
utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for secure authentication and data
protection. Even {{GSSAPI}} supports using AES, but without AES-NI support by default, so
the encryption is slow and will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the same optimization
as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be lots of
RPC calls in one connection, we needs to setup benchmark to see real improvement and then
make a trade-off. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message