hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Kumar9 <ashis...@in.ibm.com>
Subject Re: Hadoop Terasort Benchmark Failure - Need Inputs
Date Mon, 01 Dec 2014 07:18:02 GMT
I suppose you are suggesting something like below which i tried and did 
not help .

yarn jar 
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar 
  terasort -Dmapreduce.totalorderpartitioner.path =_sortPartitioning input 
output

Thanks
Ashish


From:   Bing Jiang <jiangbinglover@gmail.com>
To:     user@hadoop.apache.org
Date:   12/01/2014 11:13 AM
Subject:        Re: Hadoop Terasort Benchmark Failure - Need Inputs



hi, Ashish
I have ever seen a similar issue, and reported the issue  
https://issues.apache.org/jira/browse/MAPREDUCE-5782

I have some workaround from that jira.

-Bing



2014-11-30 4:07 GMT+08:00 Ashish Kumar9 <ashishk4@in.ibm.com>:
Hi , 

I am facing issue when i run teragen / terasort benchmark . Can someone 
suggest if you have also faced the same issue 

Command Used 
yarn jar 
/opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar 
 terasort input output 

Exception 
14/11/29 07:03:10 INFO Configuration.deprecation: 
mapred.output.value.class is deprecated. Instead, use 
mapreduce.job.output.value.class 
14/11/29 07:03:10 INFO Configuration.deprecation: 
mapred.compress.map.output is deprecated. Instead, use 
mapreduce.map.output.compress 
14/11/29 07:03:10 INFO Configuration.deprecation: 
min.num.spills.for.combine is deprecated. Instead, use 
mapreduce.map.combine.minspills 
14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001 
java.lang.IllegalArgumentException: can't read paritions file 
        at 
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)


        at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) 
        at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) 

        at 
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675) 

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740) 

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368) 
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270) 
Caused by: java.io.FileNotFoundException: File _partition.lst does not 
exist 
        at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520) 

        at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398) 

        at 
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)


        at 
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344) 
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772) 
        at 
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)


        at 
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)


        ... 6 more 
14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url 
is deprecated. Instead, use mapreduce.job.end-notification.url 
14/11/29 07:03:11 INFO mapred.JobClient:  map 0% reduce 0% 
14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001 
14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0 
14/11/29 07:03:11 INFO terasort.TeraSort: done 

Investigations done so far 
thoroughly validated mapred-site.xml and am completely in sync with below 
recommendations
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html 

teragen executes successfully but terasort fails with above exception 
some sites suggest that i should use the property "" in mapred-site.xml 
but looks like this property is no more valid . 
Granted full access to hdfs directory 
hadoop fs -chmod -R 775 /

Thanks and Regards,
Ashish Kumar




Mime
View raw message