Return-Path: Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: (qmail 15926 invoked from network); 20 Aug 2010 09:36:00 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 20 Aug 2010 09:36:00 -0000 Received: (qmail 68649 invoked by uid 500); 20 Aug 2010 09:36:00 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 68544 invoked by uid 500); 20 Aug 2010 09:35:58 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 68536 invoked by uid 99); 20 Aug 2010 09:35:57 -0000 Received: from Unknown (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 09:35:57 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 09:35:39 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7K9ZIRL015556 for ; Fri, 20 Aug 2010 09:35:18 GMT Message-ID: <8843118.466441282296918210.JavaMail.jira@thor> Date: Fri, 20 Aug 2010 05:35:18 -0400 (EDT) From: "Hong Tang (JIRA)" To: mapreduce-issues@hadoop.apache.org Subject: [jira] Commented: (MAPREDUCE-2023) TestDFSIO read test may not read specified bytes. In-Reply-To: <1636244.466241282296329542.JavaMail.jira@thor> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/MAPREDUCE-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900654#action_12900654 ] Hong Tang commented on MAPREDUCE-2023: -------------------------------------- To confirm that DFS indeed may return less than requested bytes even before reaching the end, I wrote a test program (attached), and the output of a sample run looks as follows: {noformat} hadoop dfs -ls /user/gridperf/gridmix3/part-m-00332/segment-0 Found 1 items -rw-rw-rw- 3 gridperf hdfs 1073741824 2010-08-20 08:22 /user/gridperf/gridmix3/part-m-00332/segment-0 hadoop org.apache.hadoop.fs.TestFsRead /user/gridperf/gridmix3/part-m-00332/segment-0 1000000 10995954 bytes read 21199983 bytes read 32106261 bytes read 42209617 bytes read 52456131 bytes read 63551911 bytes read 73836262 bytes read 84369397 bytes read 95182878 bytes read 105047397 bytes read 115740295 bytes read 126323360 bytes read 137166764 bytes read 147066000 bytes read 157744477 bytes read 168319334 bytes read 178856592 bytes read 188884554 bytes read 199324045 bytes read 209995098 bytes read 220916802 bytes read 231218738 bytes read 241772291 bytes read 251883835 bytes read 262306687 bytes read Fail to read a full buffer before reaching the end: pos=267640189, expected=994623, actual=795267. 272862612 bytes read 283737254 bytes read 293851212 bytes read 304525446 bytes read 314766024 bytes read 325604342 bytes read 335604768 bytes read 346475397 bytes read 357311830 bytes read 367574920 bytes read 377834612 bytes read 388029682 bytes read 398728223 bytes read 408966064 bytes read 419626247 bytes read 430260987 bytes read 440440647 bytes read 451030835 bytes read 461808645 bytes read 471996795 bytes read 482529325 bytes read 493106417 bytes read 503960340 bytes read 514155195 bytes read 524460261 bytes read 534955349 bytes read Fail to read a full buffer before reaching the end: pos=536250423, expected=999458, actual=620489. 545734170 bytes read 556326582 bytes read 567046173 bytes read 577480068 bytes read 587338410 bytes read 598115745 bytes read 608759717 bytes read 619418792 bytes read 629597629 bytes read 639906390 bytes read 650264871 bytes read 661414262 bytes read 671205472 bytes read 681856772 bytes read 692394138 bytes read 702803762 bytes read 713182701 bytes read 723720128 bytes read 734531251 bytes read 745188960 bytes read 755814801 bytes read 765670009 bytes read 776047213 bytes read 786592324 bytes read 797786600 bytes read Fail to read a full buffer before reaching the end: pos=804788073, expected=613320, actual=518295. 808158276 bytes read 818373817 bytes read 828549794 bytes read 838915719 bytes read 850189376 bytes read 860102547 bytes read 870902116 bytes read 881206170 bytes read 891441081 bytes read 902119052 bytes read 912394977 bytes read 923010497 bytes read 933330792 bytes read 944216276 bytes read 954226049 bytes read 965371734 bytes read 975663038 bytes read 986215681 bytes read 996274088 bytes read 1006954729 bytes read 1017375248 bytes read 1027801749 bytes read 1038384467 bytes read 1049383853 bytes read 1059662742 bytes read 1070106760 bytes read {noformat} > TestDFSIO read test may not read specified bytes. > ------------------------------------------------- > > Key: MAPREDUCE-2023 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2023 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks > Reporter: Hong Tang > > TestDFSIO's read test may read less bytes than specified when reading large files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.