hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bing Jiang <jiangbinglo...@gmail.com>
Subject Re: Hadoop Terasort Benchmark Failure - Need Inputs
Date Mon, 01 Dec 2014 05:45:37 GMT
hi, Ashish
I have ever seen a similar issue, and reported the issue
https://issues.apache.org/jira/browse/MAPREDUCE-5782

I have some workaround from that jira.

-Bing



2014-11-30 4:07 GMT+08:00 Ashish Kumar9 <ashishk4@in.ibm.com>:

> Hi ,
>
> I am facing issue when i run teragen / terasort benchmark . Can someone
> suggest if you have also faced the same issue
>
> *Command Used*
> yarn jar
> /opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>  terasort input output
>
> *Exception*
> 14/11/29 07:03:10 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 14/11/29 07:03:10 INFO Configuration.deprecation:
> mapred.compress.map.output is deprecated. Instead, use
> mapreduce.map.output.compress
> 14/11/29 07:03:10 INFO Configuration.deprecation:
> min.num.spills.for.combine is deprecated. Instead, use
> mapreduce.map.combine.minspills
> 14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.IllegalArgumentException: can't read paritions file
>         at
> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
>         at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
>         at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
> Caused by: java.io.FileNotFoundException: File _partition.lst does not
> exist
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
>         at
> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
>         at
> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
>         ... 6 more
> 14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url
> is deprecated. Instead, use mapreduce.job.end-notification.url
> 14/11/29 07:03:11 INFO mapred.JobClient:  map 0% reduce 0%
> 14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
> 14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
> 14/11/29 07:03:11 INFO terasort.TeraSort: done
>
> *Investigations done so far*
>
>    - thoroughly validated mapred-site.xml and am completely in sync with
>    below recommendations
>
>
> http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
>
>    - teragen executes successfully but terasort fails with above exception
>    - some sites suggest that i should use the property "" in
>    mapred-site.xml but looks like this property is no more valid .
>    - Granted full access to hdfs directory
>    hadoop fs -chmod -R 775 /
>
>
> Thanks and Regards,
> Ashish Kumar

Mime
View raw message