From common-issues-return-205960-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Sat Sep 12 21:52:04 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 58F51180660 for ; Sat, 12 Sep 2020 23:52:04 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 82F651233C6 for ; Sat, 12 Sep 2020 21:52:03 +0000 (UTC) Received: (qmail 61653 invoked by uid 500); 12 Sep 2020 21:52:03 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 61629 invoked by uid 99); 12 Sep 2020 21:52:03 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2020 21:52:03 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 3853442877 for ; Sat, 12 Sep 2020 21:52:02 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 0D072780BB1 for ; Sat, 12 Sep 2020 21:52:01 +0000 (UTC) Date: Sat, 12 Sep 2020 21:52:01 +0000 (UTC) From: "ASF GitHub Bot (Jira)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (HADOOP-17222) Create socket address leveraging URI cache MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-17222?focusedWorklogId= =3D483380&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:worklog-ta= bpanel#worklog-483380 ] ASF GitHub Bot logged work on HADOOP-17222: ------------------------------------------- Author: ASF GitHub Bot Created on: 12/Sep/20 21:51 Start Date: 12/Sep/20 21:51 Worklog Time Spent: 10m=20 Work Description: 1996fanrui commented on pull request #2241: URL: https://github.com/apache/hadoop/pull/2241#issuecomment-690855787 > I see `TestNameNodeRetryCacheMetrics.testRetryCacheMetrics` and ` Test= MultipleNNPortQOP.testMultipleNNPortOverwriteDownStream` fail on my Linux m= achine with the patch. Will confirm they are not related. >=20 > If no objections, I'll commit later this week. Thanks, @liuml07 , thanks for your test. I run these two unit tests and they are still successful. In the previou= s test of hadoop robot, TestMultipleNNPortQOP did not fail, only the last t= ime it failed. But the last time I only committed an empty, my code did not= change. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 483380) Time Spent: 4h (was: 3h 50m) > Create socket address leveraging URI cache > ------------------------------------------- > > Key: HADOOP-17222 > URL: https://issues.apache.org/jira/browse/HADOOP-17222 > Project: Hadoop Common > Issue Type: Improvement > Components: common, hdfs-client > Environment: HBase version: 2.1.0 > JVM: -Xmx2g -Xms2g=C2=A0 > hadoop hdfs version: 2.7.4 > disk:SSD > OS:CentOS Linux release 7.4.1708 (Core) > JMH Benchmark: @Fork(value =3D 1)=20 > @Warmup(iterations =3D 300)=20 > @Measurement(iterations =3D 300) > Reporter: fanrui > Assignee: fanrui > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: After Optimization remark.png, After optimization.sv= g, Before Optimization remark.png, Before optimization.svg > > Time Spent: 4h > Remaining Estimate: 0h > > Note=EF=BC=9ANot only the hdfs client can get the current benefit, all ca= llers of NetUtils.createSocketAddr will get the benefit. Just use hdfs clie= nt as an example. > =C2=A0 > Hdfs client selects best DN for hdfs Block. method call stack: > DFSInputStream.chooseDataNode -> getBestNodeDNAddrPair -> NetUtils.create= SocketAddr > NetUtils.createSocketAddr creates the corresponding InetSocketAddress bas= ed on the host and port. There are some heavier operations in the NetUtils.= createSocketAddr method, for example: URI.create(target), so NetUtils.creat= eSocketAddr takes more time to execute. > The following is my performance report. The report is based on HBase call= ing hdfs. HBase is a high-frequency access client for hdfs, because HBase r= ead operations often access a small DataBlock (about 64k) instead of the en= tire HFile. In the case of high frequency access, the NetUtils.createSocket= Addr method is time-consuming. > h3. Test Environment=EF=BC=9A > =C2=A0 > {code:java} > HBase version: 2.1.0 > JVM: -Xmx2g -Xms2g=C2=A0 > hadoop hdfs version: 2.7.4 > disk:SSD > OS:CentOS Linux release 7.4.1708 (Core) > JMH Benchmark: @Fork(value =3D 1)=20 > @Warmup(iterations =3D 300)=20 > @Measurement(iterations =3D 300) > {code} > h4. Before Optimization FlameGraph: > In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accou= nts for 4.86% of the entire CPU, and the creation of URIs accounts for a la= rger proportion. > !Before Optimization remark.png! > h3. Optimization ideas: > NetUtils.createSocketAddr creates InetSocketAddress based on host and por= t. Here we can add Cache to InetSocketAddress. The key of Cache is host and= port, and the value is InetSocketAddress. > h4. After Optimization FlameGraph: > In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accou= nts for 0.54% of the entire CPU. Here, ConcurrentHashMap is used as the Cac= he, and the ConcurrentHashMap.get() method gets data from the Cache. The CP= U usage of DFSInputStream.getBestNodeDNAddrPair has been optimized from 4.8= 6% to 0.54%. > !After Optimization remark.png! > h3. Original FlameGraph link: > [Before Optimization|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZ= LEUzGR0XAfUl6/view?usp=3Dsharing] > [After Optimization FlameGraph|https://drive.google.com/file/d/133L5m75u2= tu_KgKfGHZLEUzGR0XAfUl6/view?usp=3Dsharing] -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org