hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mostafa Mokhtar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14558) RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
Date Wed, 21 Jun 2017 01:11:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056771#comment-16056771
] 

Mostafa Mokhtar commented on HADOOP-14558:
------------------------------------------

[~hanm]
Yes, looks like a duplicate. 

> RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption

> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-14558
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14558
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.6.0
>            Reporter: Mostafa Mokhtar
>            Priority: Critical
>              Labels: impala, metadata, rpc
>
> While running a performance tests for Impala comparing secure and un-secure clusters
I noticed that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled.

> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata
loading. 
> The majority of the slowdown is coming from the two threads below. 
> {code}
> Stack Trace	Sample Count	Percentage(%)
> org.apache.hadoop.ipc.Client$Connection.run()	5,212	46.586
>    org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse()	5,203	46.505
>       java.io.DataInputStream.readInt()	5,039	45.039
>          java.io.BufferedInputStream.read()	5,038	45.03
>             java.io.BufferedInputStream.fill()	5,038	45.03
>                org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,
int)	5,036	45.013
>                   java.io.FilterInputStream.read(byte[], int, int)	5,036	45.013
>                      org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[],
int, int)	5,036	45.013
>                         org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
5,035	45.004
>                            com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int,
int)	4,775	42.68
>                               sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int,
MessageProp)	4,775	42.68
>                                  sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int,
int, MessageProp)	4,768	42.617
>                                     sun.security.jgss.krb5.WrapToken.getData()	4,714
42.134
>                                        sun.security.jgss.krb5.WrapToken.getData(byte[],
int)	4,714	42.134
>                                           sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[],
int)	4,714	42.134
>                                              sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken,
byte[], int, int, byte[], int)	3,083	27.556
>                                                 sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken,
byte[], int, int, byte[], int)	3,078	27.512
>                                                    sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[],
int, byte[], byte[], int, int)	3,076	27.494
>                                                       sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[],
int, byte[], byte[], int, int)	3,076	27.494
> {code}
> And 
> {code}
> Stack Trace	Sample Count	Percentage(%)
> java.lang.Thread.run()	3,379	30.202
>    java.util.concurrent.ThreadPoolExecutor$Worker.run()	3,379	30.202
>       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)	3,379
30.202
>          java.util.concurrent.FutureTask.run()	3,367	30.095
>             java.util.concurrent.Executors$RunnableAdapter.call()	3,367	30.095
>                org.apache.hadoop.ipc.Client$Connection$3.run()	3,367	30.095
>                   java.io.DataOutputStream.flush()	3,367	30.095
>                      java.io.BufferedOutputStream.flush()	3,367	30.095
>                         java.io.BufferedOutputStream.flushBuffer()	3,367	30.095
>                            org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[],
int, int)	3,367	30.095
>                               com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[],
int, int)	3,281	29.326
>                                  sun.security.jgss.GSSContextImpl.wrap(byte[], int, int,
MessageProp)	3,281	29.326
>                                     sun.security.jgss.krb5.Krb5Context.wrap(byte[], int,
int, MessageProp)	3,280	29.317
>                                        sun.security.jgss.krb5.WrapToken.<init>(Krb5Context,
MessageProp, byte[], int, int)	1,872	16.732
>                                        sun.security.jgss.krb5.WrapToken.encode()	1,407
12.576
> {code}
> This is the Impala Catalog thread which initiates the NameNode request
> {code}
> Stack Trace	Sample Count	Percentage(%)
> org.apache.impala.service.JniCatalog.resetMetadata(byte[])	2,414	21.577
>    org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)
2,378	21.255
>       org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)	2,378	21.255
>          org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table)	2,351
21.014
>             org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table,
boolean, boolean, Set)	2,351	21.014
>                org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient,
Set, boolean)	2,350	21.005
>                   org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List)
2,326	20.79
>                      org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor,
HdfsPartition)	2,233	19.959
>                         org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition)
1,998	17.858
>                            org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path)
1,496	13.371
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message