hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jifeng Yin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-6365) slow performance when reading big file (~10G)
Date Sat, 10 May 2014 22:15:38 GMT
Jifeng Yin created HDFS-6365:

             Summary: slow performance when reading big file (~10G)
                 Key: HDFS-6365
                 URL: https://issues.apache.org/jira/browse/HDFS-6365
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: nfs
    Affects Versions: 2.4.0
            Reporter: Jifeng Yin

Mount options:

Read a big file (~10G) ,
time dd if=/file/from/hdfs bs=4M | pv | dd of=/dev/null

First time:
1. ~160MB/s at first, without cached memory increased on the nfs server
2. ~40MB/s with cached memory increased ( cached memory is always kept).
logs show a ton of the following records:
2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_END______786375175
2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_START____803152391
2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: NFS READ fileId:
17258 offset: 3243958272 count: 65536
2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.WriteManager: No opened stream
for fileId:17258 commitOffset=3244023808. Return success in this case.

Second time:
the same as first time 2 phrase.

This message was sent by Atlassian JIRA

View raw message