hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-14558) RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
Date Wed, 21 Jun 2017 11:26:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran resolved HADOOP-14558.
-------------------------------------
    Resolution: Duplicate

Duplicate of HADOOP-10768.

Problem here is that the java encryption APIs allocate new byte buffers, get expensive fast.

If it were to be fixed, the obvious solution would be to have an option for native encryption
and a private API which would encrypt either in-place or to a preallocated buffer, the latter
being able to compensate for the fact that encrypted blocks often get their sizes rounded
up. Talk to the intel developers and see if they have time to play with this?

> RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption

> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-14558
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14558
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.6.0
>            Reporter: Mostafa Mokhtar
>            Priority: Critical
>              Labels: impala, metadata, rpc
>
> While running a performance tests for Impala comparing secure and un-secure clusters
I noticed that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled.

> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata
loading. 
> The majority of the slowdown is coming from the two threads below. 
> {code}
> Stack Trace	Sample Count	Percentage(%)
> org.apache.hadoop.ipc.Client$Connection.run()	5,212	46.586
>    org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse()	5,203	46.505
>       java.io.DataInputStream.readInt()	5,039	45.039
>          java.io.BufferedInputStream.read()	5,038	45.03
>             java.io.BufferedInputStream.fill()	5,038	45.03
>                org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,
int)	5,036	45.013
>                   java.io.FilterInputStream.read(byte[], int, int)	5,036	45.013
>                      org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[],
int, int)	5,036	45.013
>                         org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
5,035	45.004
>                            com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int,
int)	4,775	42.68
>                               sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int,
MessageProp)	4,775	42.68
>                                  sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int,
int, MessageProp)	4,768	42.617
>                                     sun.security.jgss.krb5.WrapToken.getData()	4,714
42.134
>                                        sun.security.jgss.krb5.WrapToken.getData(byte[],
int)	4,714	42.134
>                                           sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[],
int)	4,714	42.134
>                                              sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken,
byte[], int, int, byte[], int)	3,083	27.556
>                                                 sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken,
byte[], int, int, byte[], int)	3,078	27.512
>                                                    sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[],
int, byte[], byte[], int, int)	3,076	27.494
>                                                       sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[],
int, byte[], byte[], int, int)	3,076	27.494
> {code}
> And 
> {code}
> Stack Trace	Sample Count	Percentage(%)
> java.lang.Thread.run()	3,379	30.202
>    java.util.concurrent.ThreadPoolExecutor$Worker.run()	3,379	30.202
>       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)	3,379
30.202
>          java.util.concurrent.FutureTask.run()	3,367	30.095
>             java.util.concurrent.Executors$RunnableAdapter.call()	3,367	30.095
>                org.apache.hadoop.ipc.Client$Connection$3.run()	3,367	30.095
>                   java.io.DataOutputStream.flush()	3,367	30.095
>                      java.io.BufferedOutputStream.flush()	3,367	30.095
>                         java.io.BufferedOutputStream.flushBuffer()	3,367	30.095
>                            org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[],
int, int)	3,367	30.095
>                               com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[],
int, int)	3,281	29.326
>                                  sun.security.jgss.GSSContextImpl.wrap(byte[], int, int,
MessageProp)	3,281	29.326
>                                     sun.security.jgss.krb5.Krb5Context.wrap(byte[], int,
int, MessageProp)	3,280	29.317
>                                        sun.security.jgss.krb5.WrapToken.<init>(Krb5Context,
MessageProp, byte[], int, int)	1,872	16.732
>                                        sun.security.jgss.krb5.WrapToken.encode()	1,407
12.576
> {code}
> This is the Impala Catalog thread which initiates the NameNode request
> {code}
> Stack Trace	Sample Count	Percentage(%)
> org.apache.impala.service.JniCatalog.resetMetadata(byte[])	2,414	21.577
>    org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)
2,378	21.255
>       org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)	2,378	21.255
>          org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table)	2,351
21.014
>             org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table,
boolean, boolean, Set)	2,351	21.014
>                org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient,
Set, boolean)	2,350	21.005
>                   org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List)
2,326	20.79
>                      org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor,
HdfsPartition)	2,233	19.959
>                         org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition)
1,998	17.858
>                            org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path)
1,496	13.371
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message