hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jifeng Yin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-6365) slow performance when reading big file (~10G)
Date Sat, 10 May 2014 22:15:38 GMT
Jifeng Yin created HDFS-6365:
--------------------------------

             Summary: slow performance when reading big file (~10G)
                 Key: HDFS-6365
                 URL: https://issues.apache.org/jira/browse/HDFS-6365
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: nfs
    Affects Versions: 2.4.0
            Reporter: Jifeng Yin


Mount options:
{code}rw,noatime,nodiratime,vers=3,rsize=65536,wsize=65536,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys{code}

Read a big file (~10G) ,
{code}
time dd if=/file/from/hdfs bs=4M | pv | dd of=/dev/null
{code}

Results:
First time:
1. ~160MB/s at first, without cached memory increased on the nfs server
2. ~40MB/s with cached memory increased ( cached memory is always kept).
logs show a ton of the following records:
{quote}
2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_END______786375175
2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_START____803152391
2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: NFS READ fileId:
17258 offset: 3243958272 count: 65536
2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.WriteManager: No opened stream
for fileId:17258 commitOffset=3244023808. Return success in this case.
{quote}

Second time:
the same as first time 2 phrase.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message