hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HDFS-12861) Track speed in DFSClient
Date Tue, 28 Nov 2017 17:54:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269150#comment-16269150
] 

Íñigo Goiri commented on HDFS-12861:
------------------------------------

The read path is pretty straightforward (other than hedging). For the writes, we need to track
the time of sending the request and getting the ACK and then take into account the overlapping
between the badges (it may send 64 packets to write in parallel).

> Track speed in DFSClient
> ------------------------
>
>                 Key: HDFS-12861
>                 URL: https://issues.apache.org/jira/browse/HDFS-12861
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Íñigo Goiri
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to tell what
is the actual speed. We propose to add a log line with something like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502 READ 129500B
in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file /hdfs-federation/stats/2017/11/27/1511838000000.json]
hdfs.DFSClient: blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file /hdfs-federation/stats/2017/11/27/1511838000000.json]
hdfs.DFSClient: blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file /hdfs-federation/stats/2017/11/27/1511838000000.json]
hdfs.DFSClient: blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file /hdfs-federation/stats/2017/11/27/1511838000000.json]
hdfs.DFSClient: blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file /hdfs-federation/stats/2017/11/27/1511838000000.json]
hdfs.DFSClient: blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message