hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mostafa Mokhtar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-14558) RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
Date Tue, 20 Jun 2017 23:05:00 GMT
Mostafa Mokhtar created HADOOP-14558:
----------------------------------------

             Summary: RPC requests on a secure cluster are 10x slower due to expensive encryption
and decryption 
                 Key: HADOOP-14558
                 URL: https://issues.apache.org/jira/browse/HADOOP-14558
             Project: Hadoop Common
          Issue Type: Bug
          Components: hdfs-client
    Affects Versions: 2.6.0
            Reporter: Mostafa Mokhtar
            Priority: Critical


While running a performance tests for Impala comparing secure and un-secure clusters I noticed
that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled. 
hadoop.rpc.protection is set to privacy

Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata loading.


The majority of the slowdown is coming from the two threads below. 

{code}
Stack Trace	Sample Count	Percentage(%)
org.apache.hadoop.ipc.Client$Connection.run()	5,212	46.586
   org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse()	5,203	46.505
      java.io.DataInputStream.readInt()	5,039	45.039
         java.io.BufferedInputStream.read()	5,038	45.03
            java.io.BufferedInputStream.fill()	5,038	45.03
               org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int)
5,036	45.013
                  java.io.FilterInputStream.read(byte[], int, int)	5,036	45.013
                     org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[],
int, int)	5,036	45.013
                        org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
5,035	45.004
                           com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int)
4,775	42.68
                              sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int, MessageProp)
4,775	42.68
                                 sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int,
MessageProp)	4,768	42.617
                                    sun.security.jgss.krb5.WrapToken.getData()	4,714	42.134
                                       sun.security.jgss.krb5.WrapToken.getData(byte[], int)
4,714	42.134
                                          sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[],
int)	4,714	42.134
                                             sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken,
byte[], int, int, byte[], int)	3,083	27.556
                                                sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken,
byte[], int, int, byte[], int)	3,078	27.512
                                                   sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[],
int, byte[], byte[], int, int)	3,076	27.494
                                                      sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[],
int, byte[], byte[], int, int)	3,076	27.494
{code}

And 
{code}
Stack Trace	Sample Count	Percentage(%)
java.lang.Thread.run()	3,379	30.202
   java.util.concurrent.ThreadPoolExecutor$Worker.run()	3,379	30.202
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)	3,379	30.202
         java.util.concurrent.FutureTask.run()	3,367	30.095
            java.util.concurrent.Executors$RunnableAdapter.call()	3,367	30.095
               org.apache.hadoop.ipc.Client$Connection$3.run()	3,367	30.095
                  java.io.DataOutputStream.flush()	3,367	30.095
                     java.io.BufferedOutputStream.flush()	3,367	30.095
                        java.io.BufferedOutputStream.flushBuffer()	3,367	30.095
                           org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[],
int, int)	3,367	30.095
                              com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int,
int)	3,281	29.326
                                 sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp)
3,281	29.326
                                    sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int,
MessageProp)	3,280	29.317
                                       sun.security.jgss.krb5.WrapToken.<init>(Krb5Context,
MessageProp, byte[], int, int)	1,872	16.732
                                       sun.security.jgss.krb5.WrapToken.encode()	1,407	12.576
{code}

This is the Impala Catalog thread which initiates the NameNode request
{code}
Stack Trace	Sample Count	Percentage(%)
org.apache.impala.service.JniCatalog.resetMetadata(byte[])	2,414	21.577
   org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)	2,378
21.255
      org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)	2,378	21.255
         org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table)	2,351
21.014
            org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table, boolean,
boolean, Set)	2,351	21.014
               org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient,
Set, boolean)	2,350	21.005
                  org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List)	2,326
20.79
                     org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor,
HdfsPartition)	2,233	19.959
                        org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition)
1,998	17.858
                           org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path)	1,496
13.371
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message