hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14541) ShortCircuitReplica#unref cost about 6% cpu and 6% heap allocation because of the frequent thrown NoSuchElementException in our HBase benchmark
Date Tue, 18 Jun 2019 10:42:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866473#comment-16866473
] 

Hadoop QA commented on HDFS-14541:
----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  0s{color} | {color:blue}
Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  6s{color} | {color:red}
HDFS-14541 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14541 |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26991/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This message was automatically generated.



> ShortCircuitReplica#unref cost about 6% cpu and 6% heap allocation because of the frequent
thrown NoSuchElementException  in our HBase benchmark
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-14541
>                 URL: https://issues.apache.org/jira/browse/HDFS-14541
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Zheng Hu
>            Assignee: Lisheng Sun
>            Priority: Major
>         Attachments: HDFS-14541.000.patch, after-QPS.png, after-cpu-flame-graph.svg,
after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg, async-prof-pid-94152-cpu-1.svg,
before-QPS.png, before-cpu-flame-graph.svg, before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of HBASE-21879,  and
we have few CPU flame graph  & heap flame graph by using async-profiler,  and find that
there're some performence issues in DFSClient  . 
> See the attached two flame graphs, we can conclude that the try catch block in ShortCircuitCache#trimEvictionMaps
 has some serious perf problem , we should remove the try catch from DFSClient. 
> {code}
>   /**
>    * Trim the eviction lists.
>    */
>   private void trimEvictionMaps() {
>     long now = Time.monotonicNow();
>     demoteOldEvictableMmaped(now);
>     while (true) {
>       long evictableSize = evictable.size();
>       long evictableMmappedSize = evictableMmapped.size();
>       if (evictableSize + evictableMmappedSize <= maxTotalSize) {
>         return;
>       }
>       ShortCircuitReplica replica;
>       try {
>         if (evictableSize == 0) {
>           replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
>               .firstKey());
>         } else {
>           replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
>         }
>       } catch (NoSuchElementException e) {
>         break;
>       }
>       if (LOG.isTraceEnabled()) {
>         LOG.trace(this + ": trimEvictionMaps is purging " + replica +
>             StringUtils.getStackTrace(Thread.currentThread()));
>       }
>       purge(replica);
>     }
>   }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message