hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ling Kun <lkun.e...@gmail.com>
Subject Re: error while running TestDFSIO
Date Sat, 20 Apr 2013 01:14:48 GMT
I have got the same problem. And also need help.

Ling Kun


On Sat, Apr 20, 2013 at 8:35 AM, kaveh minooie <kaveh@plutoz.com> wrote:

> Hi everyone
>
> I am getting this error when i run TestDFSIO. the job actually finishes
> successfully. ( according to jobtracker at least ) but this is what i get
> on the console :
>
> crawler@d1r2n2:/hadoop$ bin/hadoop jar hadoop-test-1.1.1.jar  TestDFSIO
> -write -nrFiles 10 -fileSize 1000
> TestDFSIO.0.0.4
> 13/04/19 17:23:43 INFO fs.TestDFSIO: nrFiles = 10
> 13/04/19 17:23:43 INFO fs.TestDFSIO: fileSize (MB) = 1000
> 13/04/19 17:23:43 INFO fs.TestDFSIO: bufferSize = 1000000
> 13/04/19 17:23:43 INFO fs.TestDFSIO: creating control file: 1000 mega
> bytes, 10 files
> 13/04/19 17:23:44 INFO fs.TestDFSIO: created control files for: 10 files
> 13/04/19 17:23:44 INFO mapred.FileInputFormat: Total input paths to
> process : 10
> 13/04/19 17:23:44 INFO mapred.JobClient: Running job: job_201304191712_0002
> 13/04/19 17:23:45 INFO mapred.JobClient:  map 0% reduce 0%
> 13/04/19 17:24:06 INFO mapred.JobClient:  map 20% reduce 0%
> 13/04/19 17:24:07 INFO mapred.JobClient:  map 30% reduce 0%
> 13/04/19 17:24:09 INFO mapred.JobClient:  map 50% reduce 0%
> 13/04/19 17:24:11 INFO mapred.JobClient:  map 60% reduce 0%
> 13/04/19 17:24:12 INFO mapred.JobClient:  map 90% reduce 0%
> 13/04/19 17:24:13 INFO mapred.JobClient:  map 100% reduce 0%
> 13/04/19 17:24:21 INFO mapred.JobClient:  map 100% reduce 33%
> 13/04/19 17:24:22 INFO mapred.JobClient:  map 100% reduce 100%
> 13/04/19 17:24:23 INFO mapred.JobClient: Job complete:
> job_201304191712_0002
> 13/04/19 17:24:23 INFO mapred.JobClient: Counters: 33
> 13/04/19 17:24:23 INFO mapred.JobClient:   Job Counters
> 13/04/19 17:24:23 INFO mapred.JobClient:     Launched reduce tasks=1
> 13/04/19 17:24:23 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=210932
> 13/04/19 17:24:23 INFO mapred.JobClient:     Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/04/19 17:24:23 INFO mapred.JobClient:     Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/04/19 17:24:23 INFO mapred.JobClient:     Rack-local map tasks=2
> 13/04/19 17:24:23 INFO mapred.JobClient:     Launched map tasks=10
> 13/04/19 17:24:23 INFO mapred.JobClient:     Data-local map tasks=8
> 13/04/19 17:24:23 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=8650
> 13/04/19 17:24:23 INFO mapred.JobClient:   File Input Format Counters
> 13/04/19 17:24:23 INFO mapred.JobClient:     Bytes Read=1120
> 13/04/19 17:24:23 INFO mapred.JobClient:   SkippingTaskCounters
> 13/04/19 17:24:23 INFO mapred.JobClient:     MapProcessedRecords=10
> 13/04/19 17:24:23 INFO mapred.JobClient:     ReduceProcessedGroups=5
> 13/04/19 17:24:23 INFO mapred.JobClient:   File Output Format Counters
> 13/04/19 17:24:23 INFO mapred.JobClient:     Bytes Written=79
> 13/04/19 17:24:23 INFO mapred.JobClient:   FileSystemCounters
> 13/04/19 17:24:23 INFO mapred.JobClient:     FILE_BYTES_READ=871
> 13/04/19 17:24:23 INFO mapred.JobClient:     HDFS_BYTES_READ=2330
> 13/04/19 17:24:23 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=272508
> 13/04/19 17:24:23 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=10485760079
> 13/04/19 17:24:23 INFO mapred.JobClient:   Map-Reduce Framework
> 13/04/19 17:24:23 INFO mapred.JobClient:     Map output materialized
> bytes=925
> 13/04/19 17:24:23 INFO mapred.JobClient:     Map input records=10
> 13/04/19 17:24:23 INFO mapred.JobClient:     Reduce shuffle bytes=925
> 13/04/19 17:24:23 INFO mapred.JobClient:     Spilled Records=100
> 13/04/19 17:24:23 INFO mapred.JobClient:     Map output bytes=765
> 13/04/19 17:24:23 INFO mapred.JobClient:     Total committed heap usage
> (bytes)=7996702720
> 13/04/19 17:24:23 INFO mapred.JobClient:     CPU time spent (ms)=104520
> 13/04/19 17:24:23 INFO mapred.JobClient:     Map input bytes=260
> 13/04/19 17:24:23 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1210
> 13/04/19 17:24:23 INFO mapred.JobClient:     Combine input records=0
> 13/04/19 17:24:23 INFO mapred.JobClient:     Reduce input records=50
> 13/04/19 17:24:23 INFO mapred.JobClient:     Reduce input groups=5
> 13/04/19 17:24:23 INFO mapred.JobClient:     Combine output records=0
> 13/04/19 17:24:23 INFO mapred.JobClient:     Physical memory (bytes)
> snapshot=7111999488
> 13/04/19 17:24:23 INFO mapred.JobClient:     Reduce output records=5
> 13/04/19 17:24:23 INFO mapred.JobClient:     Virtual memory (bytes)
> snapshot=28466053120
> 13/04/19 17:24:23 INFO mapred.JobClient:     Map output records=50
> java.io.FileNotFoundException: File does not exist:
> /benchmarks/TestDFSIO/io_**write/part-00000
>         at org.apache.hadoop.hdfs.**DFSClient$DFSInputStream.**
> fetchLocatedBlocks(DFSClient.**java:1975)
>         at org.apache.hadoop.hdfs.**DFSClient$DFSInputStream.**
> openInfo(DFSClient.java:1944)
>         at org.apache.hadoop.hdfs.**DFSClient$DFSInputStream.<**
> init>(DFSClient.java:1936)
>         at org.apache.hadoop.hdfs.**DFSClient.open(DFSClient.java:**731)
>         at org.apache.hadoop.hdfs.**DistributedFileSystem.open(**
> DistributedFileSystem.java:**165)
>         at org.apache.hadoop.fs.**FileSystem.open(FileSystem.**java:427)
>         at org.apache.hadoop.fs.**TestDFSIO.analyzeResult(**
> TestDFSIO.java:339)
>         at org.apache.hadoop.fs.**TestDFSIO.run(TestDFSIO.java:**462)
>         at org.apache.hadoop.util.**ToolRunner.run(ToolRunner.**java:65)
>         at org.apache.hadoop.util.**ToolRunner.run(ToolRunner.**java:79)
>         at org.apache.hadoop.fs.**TestDFSIO.main(TestDFSIO.java:**317)
>         at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>         at sun.reflect.**NativeMethodAccessorImpl.**invoke(Unknown Source)
>         at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(Unknown
> Source)
>         at java.lang.reflect.Method.**invoke(Unknown Source)
>         at org.apache.hadoop.util.**ProgramDriver$**
> ProgramDescription.invoke(**ProgramDriver.java:68)
>         at org.apache.hadoop.util.**ProgramDriver.driver(**
> ProgramDriver.java:139)
>         at org.apache.hadoop.test.**AllTestDriver.main(**
> AllTestDriver.java:81)
>         at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method)
>         at sun.reflect.**NativeMethodAccessorImpl.**invoke(Unknown Source)
>         at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(Unknown
> Source)
>         at java.lang.reflect.Method.**invoke(Unknown Source)
>         at org.apache.hadoop.util.RunJar.**main(RunJar.java:156)
> crawler@d1r2n2:/hadoop$ bin/hadoop fs -ls /benchmarks/TestDFSIO/io_write
> Found 3 items
> -rw-r--r--   2 crawler supergroup          0 2013-04-19 17:24
> /benchmarks/TestDFSIO/io_**write/_SUCCESS
> drwxr-xr-x   - crawler supergroup          0 2013-04-19 17:23
> /benchmarks/TestDFSIO/io_**write/_logs
> -rw-r--r--   2 crawler supergroup         79 2013-04-19 17:24
> /benchmarks/TestDFSIO/io_**write/part-00000.deflate
> crawler@d1r2n2:/hadoop$
>
>
> Does anyone have any idea what might be wrong here?
>
>


-- 
http://www.lingcc.com

Mime
View raw message