hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Kumar9 <ashis...@in.ibm.com>
Subject Hadoop Terasort Benchmark Failure - Need Inputs
Date Sat, 29 Nov 2014 20:07:01 GMT
Hi ,

I am facing issue when i run teragen / terasort benchmark . Can someone 
suggest if you have also faced the same issue 

Command Used
yarn jar 
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar 
 terasort input output

Exception
14/11/29 07:03:10 INFO Configuration.deprecation: 
mapred.output.value.class is deprecated. Instead, use 
mapreduce.job.output.value.class
14/11/29 07:03:10 INFO Configuration.deprecation: 
mapred.compress.map.output is deprecated. Instead, use 
mapreduce.map.output.compress
14/11/29 07:03:10 INFO Configuration.deprecation: 
min.num.spills.for.combine is deprecated. Instead, use 
mapreduce.map.combine.minspills
14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.IllegalArgumentException: can't read paritions file
        at 
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
        at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at 
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
Caused by: java.io.FileNotFoundException: File _partition.lst does not 
exist
        at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
        at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
        at 
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
        at 
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
        at 
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
        at 
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
        ... 6 more
14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url 
is deprecated. Instead, use mapreduce.job.end-notification.url
14/11/29 07:03:11 INFO mapred.JobClient:  map 0% reduce 0%
14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
14/11/29 07:03:11 INFO terasort.TeraSort: done

Investigations done so far
thoroughly validated mapred-site.xml and am completely in sync with below 
recommendations
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
teragen executes successfully but terasort fails with above exception
some sites suggest that i should use the property "" in mapred-site.xml 
but looks like this property is no more valid . 
Granted full access to hdfs directory 
hadoop fs -chmod -R 775 /

Thanks and Regards,
Ashish Kumar
Mime
View raw message