Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Date: Fri, 14 Oct 2011 19:50:11 +0000 (UTC)
From: "John George (Resolved) (JIRA)" <jira@apache.org>
To: mapreduce-dev@hadoop.apache.org
Message-ID: 
 <686346346.15341.1318621811888.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1024389057.2142.1317111132519.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (MAPREDUCE-3107) DFSIO tests are failing
 intermittently
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John George resolved MAPREDUCE-3107.
------------------------------------

    Resolution: Cannot Reproduce

It seems like this was fixed somehow. I confirmed with Rajit and he does not see it anymore as well. Hence, closing this as cannot be reproduced.
                
> DFSIO tests are failing intermittently
> --------------------------------------
>
>                 Key: MAPREDUCE-3107
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3107
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: benchmarks, mrv2
>    Affects Versions: 0.23.0
>            Reporter: Rajit Saha
>            Assignee: John George
>            Priority: Blocker
>             Fix For: 0.23.0
>
>
> Intermittently DFSIO tests are failing either in "read" or "write" operations.
> $HADOOP_COMMON_HOME/bin/hadoop --config $HADOOP_CONF_DIR jar $HADOOP_MAPRED_HOME/hadoop-mapreduce-test-*.jar TestDFSIO  -read -nrFiles 36 -fileSize 320
> 11/09/27 03:21:03 INFO fs.TestDFSIO: TestDFSIO.0.0.6
> 11/09/27 03:21:03 INFO fs.TestDFSIO: nrFiles = 36
> 11/09/27 03:21:03 INFO fs.TestDFSIO: fileSize (MB) = 320.0
> 11/09/27 03:21:03 INFO fs.TestDFSIO: bufferSize = 1000000
> 11/09/27 03:21:03 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
> 11/09/27 03:21:03 INFO fs.TestDFSIO: creating control file: 335544320 bytes, 36 files
> 11/09/27 03:21:05 INFO fs.TestDFSIO: created control files for: 36 files
> ..
> ..
> ..
> ..
> ..
> 11/09/27 03:24:04 INFO mapreduce.Job:  map 88% reduce 25%
> 11/09/27 03:24:05 INFO mapreduce.Job:  map 90% reduce 25%
> 11/09/27 03:24:06 INFO mapreduce.Job:  map 92% reduce 25%
> 11/09/27 03:24:09 INFO mapreduce.Job:  map 93% reduce 25%
> 11/09/27 03:24:10 INFO mapreduce.Job:  map 93% reduce 26%
> 11/09/27 03:24:42 INFO mapreduce.Job:  map 94% reduce 26%
> 11/09/27 03:24:43 INFO mapreduce.Job:  map 94% reduce 27%
> 11/09/27 03:24:56 INFO mapreduce.Job:  map 95% reduce 27%
> 11/09/27 03:24:58 INFO mapreduce.Job:  map 95% reduce 28%
> 11/09/27 03:25:13 INFO mapreduce.Job:  map 96% reduce 28%
> 11/09/27 03:25:15 INFO mapreduce.Job:  map 97% reduce 28%
> 11/09/27 03:25:16 INFO mapreduce.Job:  map 97% reduce 30%
> 11/09/27 03:25:16 INFO mapreduce.Job:  map 98% reduce 30%
> 11/09/27 03:25:16 INFO mapreduce.Job: Job job_1317092846056_0012 failed with state FAILED
> 11/09/27 03:25:16 INFO mapreduce.Job: Counters: 44
> 	File System Counters
> 		FILE: BYTES_READ=20931
> 		FILE: BYTES_WRITTEN=2192966
> 		FILE: READ_OPS=0
> 		FILE: LARGE_READ_OPS=0
> 		FILE: WRITE_OPS=0
> 		HDFS: BYTES_READ=10454250228
> 		HDFS: BYTES_WRITTEN=0
> 		HDFS: READ_OPS=172
> 		HDFS: LARGE_READ_OPS=0
> 		HDFS: WRITE_OPS=0
> 	org.apache.hadoop.mapreduce.JobCounter
> 		NUM_FAILED_MAPS=6
> 		TOTAL_LAUNCHED_MAPS=44
> 		TOTAL_LAUNCHED_REDUCES=1
> 		DATA_LOCAL_MAPS=10
> 		RACK_LOCAL_MAPS=34
> 		SLOTS_MILLIS_MAPS=2340307
> 	org.apache.hadoop.mapreduce.TaskCounter
> 		MAP_INPUT_RECORDS=36
> 		MAP_OUTPUT_RECORDS=140
> 		MAP_OUTPUT_BYTES=2103
> 		MAP_OUTPUT_MATERIALIZED_BYTES=2579
> 		SPLIT_RAW_BYTES=5210
> 		COMBINE_INPUT_RECORDS=0
> 		COMBINE_OUTPUT_RECORDS=0
> 		REDUCE_INPUT_GROUPS=0
> 		REDUCE_SHUFFLE_BYTES=3054
> 		REDUCE_INPUT_RECORDS=0
> 		REDUCE_OUTPUT_RECORDS=0
> 		SPILLED_RECORDS=140
> 		SHUFFLED_MAPS=33
> 		FAILED_SHUFFLE=0
> 		MERGED_MAP_OUTPUTS=0
> 		GC_TIME_MILLIS=59933
> 		CPU_MILLISECONDS=159470
> 		PHYSICAL_MEMORY_BYTES=11310596096
> 		VIRTUAL_MEMORY_BYTES=31425290240
> 		COMMITTED_HEAP_BYTES=12728664064
> 	Shuffle Errors
> 		BAD_ID=0
> 		CONNECTION=0
> 		IO_ERROR=0
> 		WRONG_LENGTH=0
> 		WRONG_MAP=0
> 		WRONG_REDUCE=0
> 	org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
> 		BYTES_READ=4058
> 	org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter
> 		BYTES_WRITTEN=0
> java.io.IOException: Job failed!
> 	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:781)
> 	at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:340)
> 	at org.apache.hadoop.fs.TestDFSIO.readTest(TestDFSIO.java:418)
> 	at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:522)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> 	at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:445)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> 	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> 	at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> 	at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:189)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira