hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wellington Chevreuil (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
Date Tue, 08 Dec 2015 17:31:11 GMT
Wellington Chevreuil created HDFS-9521:
------------------------------------------

             Summary: TransferFsImage.receiveFile should account and log separate times for
image download and fsync to disk 
                 Key: HDFS-9521
                 URL: https://issues.apache.org/jira/browse/HDFS-9521
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Wellington Chevreuil
            Assignee: Wellington Chevreuil


Currently, TransferFsImage.receiveFile is logging total transfer time as below:

{noformat}
double xferSec = Math.max(
       ((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);    
long xferKb = received / 1024;
LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb / xferSec))
{noformat}

This is really useful, but it just measures the total method execution time, which includes
time taken to download the image and do an fsync to all the namenode metadata directories.

Sometime when troubleshooting these imager transfer problems, it's interesting to know which
part of the process is being the bottleneck (whether network or disk write).

This patch accounts time for image download and fsync to each disk separately, logging how
much time did it take on each operation.


 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message