Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4CD1210881 for ; Sat, 29 Nov 2014 20:11:24 +0000 (UTC) Received: (qmail 17288 invoked by uid 500); 29 Nov 2014 20:11:19 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 17171 invoked by uid 500); 29 Nov 2014 20:11:19 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 17161 invoked by uid 99); 29 Nov 2014 20:11:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Nov 2014 20:11:19 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ashishk4@in.ibm.com designates 122.248.162.7 as permitted sender) Received: from [122.248.162.7] (HELO e28smtp07.in.ibm.com) (122.248.162.7) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Nov 2014 20:11:11 +0000 Received: from /spool/local by e28smtp07.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sun, 30 Nov 2014 01:39:48 +0530 Received: from d28dlp01.in.ibm.com (9.184.220.126) by e28smtp07.in.ibm.com (192.168.1.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sun, 30 Nov 2014 01:39:47 +0530 Received: from d28relay03.in.ibm.com (d28relay03.in.ibm.com [9.184.220.60]) by d28dlp01.in.ibm.com (Postfix) with ESMTP id DB601E003F for ; Sun, 30 Nov 2014 01:40:12 +0530 (IST) Received: from d28av06.in.ibm.com (d28av06.in.ibm.com [9.184.220.48]) by d28relay03.in.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id sATKBuYT3408302 for ; Sun, 30 Nov 2014 01:41:56 +0530 Received: from d28av06.in.ibm.com (localhost [127.0.0.1]) by d28av06.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id sATK9lFT031431 for ; Sun, 30 Nov 2014 01:39:47 +0530 Received: from d23ml193.in.ibm.com (d23ml193.in.ibm.com [9.182.8.45]) by d28av06.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id sATK9kCH031428 for ; Sun, 30 Nov 2014 01:39:46 +0530 To: user@hadoop.apache.org MIME-Version: 1.0 Subject: Hadoop Terasort Benchmark Failure - Need Inputs X-KeepSent: 0CA3DA3D:A157B73C-65257D9F:006EA0F4; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1SHF211 December 19, 2013 From: Ashish Kumar9 Message-ID: Date: Sun, 30 Nov 2014 01:37:01 +0530 X-MIMETrack: Serialize by Router on d23ml193/23/M/IBM(Release 8.5.3FP6|November 21, 2013) at 30/11/2014 01:37:02, Serialize complete at 30/11/2014 01:37:02 Content-Type: multipart/alternative; boundary="=_alternative 006EC1CE65257D9F_=" X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14112920-0025-0000-0000-000002250280 X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. --=_alternative 006EC1CE65257D9F_= Content-Type: text/plain; charset="US-ASCII" Hi , I am facing issue when i run teragen / terasort benchmark . Can someone suggest if you have also faced the same issue Command Used yarn jar /opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar terasort input output Exception 14/11/29 07:03:10 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/11/29 07:03:10 INFO Configuration.deprecation: mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress 14/11/29 07:03:10 INFO Configuration.deprecation: min.num.spills.for.combine is deprecated. Instead, use mapreduce.map.combine.minspills 14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001 java.lang.IllegalArgumentException: can't read paritions file at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:675) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270) Caused by: java.io.FileNotFoundException: File _partition.lst does not exist at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:142) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772) at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158) at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213) ... 6 more 14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url is deprecated. Instead, use mapreduce.job.end-notification.url 14/11/29 07:03:11 INFO mapred.JobClient: map 0% reduce 0% 14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001 14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0 14/11/29 07:03:11 INFO terasort.TeraSort: done Investigations done so far thoroughly validated mapred-site.xml and am completely in sync with below recommendations http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html teragen executes successfully but terasort fails with above exception some sites suggest that i should use the property "" in mapred-site.xml but looks like this property is no more valid . Granted full access to hdfs directory hadoop fs -chmod -R 775 / Thanks and Regards, Ashish Kumar --=_alternative 006EC1CE65257D9F_= Content-Type: text/html; charset="US-ASCII" Hi ,

I am facing issue when i run teragen / terasort benchmark . Can someone suggest if you have also faced the same issue

Command Used
yarn jar /opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar  terasort input output

Exception
14/11/29 07:03:10 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/11/29 07:03:10 INFO Configuration.deprecation: mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress
14/11/29 07:03:10 INFO Configuration.deprecation: min.num.spills.for.combine is deprecated. Instead, use mapreduce.map.combine.minspills
14/11/29 07:03:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.IllegalArgumentException: can't read paritions file
        at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:216)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:270)
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:772)
        at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:158)
        at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:213)
        ... 6 more
14/11/29 07:03:10 INFO Configuration.deprecation: job.end.notification.url is deprecated. Instead, use mapreduce.job.end-notification.url
14/11/29 07:03:11 INFO mapred.JobClient:  map 0% reduce 0%
14/11/29 07:03:11 INFO mapred.JobClient: Job complete: job_local_0001
14/11/29 07:03:11 INFO mapred.JobClient: Counters: 0
14/11/29 07:03:11 INFO terasort.TeraSort: done

Investigations done so far
  • thoroughly validated mapred-site.xml and am completely in sync with below recommendations
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
  • teragen executes successfully but terasort fails with above exception
  • some sites suggest that i should use the property "" in mapred-site.xml but looks like this property is no more valid .
  • Granted full access to hdfs directory
    hadoop fs -chmod -R 775 /

Thanks and Regards,
Ashish Kumar --=_alternative 006EC1CE65257D9F_=--