hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Dvorzhak (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
Date Fri, 22 Dec 2017 06:16:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300770#comment-16300770
] 

Igor Dvorzhak edited comment on HADOOP-15124 at 12/22/17 6:15 AM:
------------------------------------------------------------------

I have added a draft implementation of a property that disables per-thread statistics to the
PR.

Here is performance profiling results of {{Statistics}} code paths with latest changes:
|| || || ||TeraGen 1TB|| || || ||
|| || ||GCS|| || ||HDFS|| ||
| |No changes|LongAdder + Per-Thread|LongAdder|No changes|LongAdder + Per-Thread|LongAdder|
|CPU, ms|1,430|1,880|3.86|805.85|349.64|172.6|
|CPU, %|23.4|28.4|0.057|10.9|4.61|2.29|
|Wall, ms|11,420|1,900|3.79|11,270|503.7|242.83|
|Wall, %|6.45|1.11|0.002|6.34|0.303|0.146|
Legend:
* "No changes" - stock Hadoop
* "LongAdder + Per-Thread" - FS.Statistics with LongAdder backend and with per-thread statistic
enabled
* "LongAdder" - FS.Statistics with LongAdder backend and with per-thread statistic disabled

Performance profiling results summary: this change improves performance in all tested cases
except slight regression in CPU time for GCS with enabled per-thread statistics.

While change is not complete yet (needs tests, javadocs and proper property wiring), main
functionality looks ready for review already.

What should I do to move this change forward? (I think it will be better to try to get consensus
on it, before investing more time into finishing it, in case it will be rejected).

Also, I didn't figure out yet how to pass configuration/property value to FileSystem.Statistics
class. Maybe you can advise here?


was (Author: medb):
I have added a draft implementation of a property that disables per-thread statistics to the
PR.

Here is performance profiling results of {{Statistics}} code paths with latest changes:
|| || || ||TeraGen 1TB|| || || ||
|| || ||GCS|| || ||HDFS|| ||
| |No changes|LongAdder + Per-Thread|LongAdder|No changes|LongAdder + Per-Thread|LongAdder|
|CPU, ms|1,430|1,880|3.86|805.85|349.64|172.6|
|CPU, %|23.4|28.4|0.057|10.9|4.61|2.29|
|Wall, ms|11,420|1,900|3.79|11,270|503.7|242.83|
|Wall, %|6.45|1.11|0.002|6.34|0.303|0.146|
Legend:
* "No changes" - stock Hadoop
* "LongAdder + Per-Thread" - FS.Statistics with LongAdder backend and with per-thread statistic
enabled
* "LongAdder" - FS.Statistics with LongAdder backend and with per-thread statistic disabled

Performance profiling results summary: this change improves performance in all tested cases
except slight regression in CPU time for GCS with enabled per-thread statistics.

While change is not complete yet (needs tests, javadocs and proper property wiring), main
functipnaity looks ready for review already.

What should I do to move this change forward? (I think it will be better to try to get consensus
on it, before investing more time into finishing it, in case it will be rejected).

Also, I didn't figure out yet how to pass configuration/property value to FileSystem.Statistics
class. Maybe you can advise here?

> Slow FileSystem.Statistics counters implementation
> --------------------------------------------------
>
>                 Key: HADOOP-15124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15124
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: common
>    Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0
>            Reporter: Igor Dvorzhak
>            Assignee: Igor Dvorzhak
>              Labels: common, filesystem, statistics
>
> While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 workers,
GCS connector) I saw that FileSystem.Statistics code paths Wall time is 5.58% and CPU time
is 26.5% of total execution time.
> After switching FileSystem.Statistics implementation to LongAdder, consumed Wall time
decreased to 0.006% and CPU time to 0.104% of total execution time.
> Total job runtime decreased from 66 mins to 61 mins.
> These results are not conclusive, because I didn't benchmark multiple times to average
results, but regardless of performance gains switching to LongAdder simplifies code and reduces
its complexity.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message