Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6807818606 for ; Wed, 27 May 2015 12:10:41 +0000 (UTC) Received: (qmail 71827 invoked by uid 500); 27 May 2015 12:10:30 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 71709 invoked by uid 500); 27 May 2015 12:10:30 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 71699 invoked by uid 99); 27 May 2015 12:10:30 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 May 2015 12:10:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id CABA01A35A4 for ; Wed, 27 May 2015 12:10:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.9 X-Spam-Level: *** X-Spam-Status: No, score=3.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_LIVE=1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 2ayPEnczDYD1 for ; Wed, 27 May 2015 12:10:20 +0000 (UTC) Received: from mail-qc0-f179.google.com (mail-qc0-f179.google.com [209.85.216.179]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 0C79D210AE for ; Wed, 27 May 2015 12:10:20 +0000 (UTC) Received: by qcxw10 with SMTP id w10so2695112qcx.3 for ; Wed, 27 May 2015 05:09:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=aMQwUSbMGeWqWoDpiwBwdjdR7FC1IT2GAjEXgnFhh1A=; b=Du0CvP2u4nYtoTSghR27HiwKQ0r88cREpBrA7XFpyC499fJ2KYoRNurKqclWSljz/r syLQ3kX/FDA/jmBAT4keWAf4W5oVYwvPveRtCMIV7tXojWylvUj7PD8/+DTH9aoP8G7m FhG80G7lAVXF48lkfYyJtmu/JSU+CwPLOTwGIfT+vUMoS6rnxUxkiBpDtLtKoUKjPvsn TfVPx6hxBQargt6oKHNxeSjIRM4+k2H6x8TSttqTrQrzzjKTKfR2P8EJ2d6DGnHb898/ jKhNJncb8B6cZuDYJzvFzSwdCGiZcNAZbR/2PBN6tS40BehXSzLhnbZAd8OxBp0O2E91 6E6w== X-Received: by 10.140.133.9 with SMTP id 9mr36127207qhf.5.1432728574355; Wed, 27 May 2015 05:09:34 -0700 (PDT) Received: from [172.20.3.112] ([67.201.113.11]) by mx.google.com with ESMTPSA id u10sm10131325qgd.14.2015.05.27.05.09.32 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 27 May 2015 05:09:32 -0700 (PDT) From: Alexander Alten-Lorenz Content-Type: multipart/alternative; boundary="Apple-Mail=_AA896689-93DA-4FE4-AE5C-EAF174C5CF9F" Message-Id: <436D566F-426D-4466-B1C8-D7720ED90D03@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2100\)) Subject: Re: Teragen-Terasort>=10GB fails Date: Wed, 27 May 2015 08:09:31 -0400 References: To: user@hadoop.apache.org In-Reply-To: X-Mailer: Apple Mail (2.2100) --Apple-Mail=_AA896689-93DA-4FE4-AE5C-EAF174C5CF9F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii FSError: java.io.IOException: No space left on device check the local disks for enough free tmp space in /hadoop/ best, -- Alexander Alten-Lorenz m: wget.null@gmail.com b: mapredit.blogspot.com > On 27 May 2015, at 6:29 am, Pratik Gadiya = wrote: >=20 > =20 > Hi All, > =20 > When I run teragen-terasort test on my hadoop deployed cluster, I get = following error > =20 > 15/05/27 06:24:36 INFO mapreduce.Job: map 57% reduce 18% > 15/05/27 06:24:39 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_r_000000_0, Status : FAILED > Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: = error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled = map-outputs > at = org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1628) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: = Could not find any valid local directory for = output/attempt_1432720271082_0005_r_000000_0/map_38.out > at = org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathFor= Write(LocalDirAllocator.java:402) > at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:150) > at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:131) > at = org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFi= les.java:213) > at = org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.me= rge(MergeManagerImpl.java:457) > at = org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:9= 4) >=20 > 15/05/27 06:24:40 INFO mapreduce.Job: map 57% reduce 0% > 15/05/27 06:24:46 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000041_0, Status : FAILED > FSError: java.io.IOException: No space left on device > 15/05/27 06:24:48 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000046_0, Status : FAILED > FSError: java.io.IOException: No space left on device > 15/05/27 06:24:49 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000044_0, Status : FAILED > Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could = not find any valid local directory for = attempt_1432720271082_0005_m_000044_0_spill_0.out > at = org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathFor= Write(LocalDirAllocator.java:402) > at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:150) > at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:131) > at = org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFi= les.java:159) > at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java= :1584) > at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at = org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:720= ) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1628) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >=20 > 15/05/27 06:24:50 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000045_0, Status : FAILED > Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could = not find any valid local directory for = attempt_1432720271082_0005_m_000045_0_spill_0.out > at = org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathFor= Write(LocalDirAllocator.java:402) > at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:150) > at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:131) > at = org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFi= les.java:159) > at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java= :1584) > at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at = org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:720= ) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1628) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >=20 > 15/05/27 06:24:51 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000041_1, Status : FAILED > mkdir of = /hadoop/yarn/local/usercache/hdfs/appcache/application_1432720271082_0005/= container_1432720271082_0005_01_000050 failed >=20 > 15/05/27 06:24:54 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000046_1, Status : FAILED > FSError: java.io.IOException: No space left on device > =20 > Can someone let me know how can I resolve this error ? > =20 > With Regards, > Pratik Gadiya > =20 > DISCLAIMER =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e-mail may contain = privileged and confidential information which is the property of = Persistent Systems Ltd. It is intended only for the use of the = individual or entity to which it is addressed. If you are not the = intended recipient, you are not authorized to read, retain, copy, print, = distribute or use this message. If you have received this communication = in error, please notify the sender and delete all copies of this = message. Persistent Systems Ltd. does not accept any liability for virus = infected mails. >=20 --Apple-Mail=_AA896689-93DA-4FE4-AE5C-EAF174C5CF9F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
FSError: = java.io.IOException: No space left on device

check the local disks for enough free tmp = space in /hadoop/

best,

--
Alexander Alten-Lorenz
m: wget.null@gmail.com
b: mapredit.blogspot.com

On 27 May 2015, at 6:29 am, Pratik Gadiya <pratik_gadiya@persistent.com> wrote:

 
Hi = All,
 
When I = run teragen-terasort test on my hadoop deployed cluster, I get following = error
 
15/05/27 06:24:36 INFO mapreduce.Job: map 57% = reduce 18%
15/05/27 06:24:39 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_r_000000_0, Status : FAILED
Error: = org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in = shuffle in InMemoryMerger - Thread to merge in-memory shuffled = map-outputs
at = org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at = org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at = org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native = Method)
at = javax.security.auth.Subject.doAs(Subject.java:415)
at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1628)
at = org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: = org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find = any valid local directory for = output/attempt_1432720271082_0005_r_000000_0/map_38.out
at = org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathFor= Write(LocalDirAllocator.java:402)
at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:150)
at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:131)
at = org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFi= les.java:213)
at = org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.me= rge(MergeManagerImpl.java:457)
at = org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:9= 4)

15/05/27 06:24:40 INFO mapreduce.Job: = map 57% reduce 0%
15/05/27 06:24:46 INFO mapreduce.Job: = Task Id : attempt_1432720271082_0005_m_000041_0, Status : FAILED
FSError: java.io.IOException: No space left on device
15/05/27 06:24:48 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000046_0, Status : FAILED
FSError: java.io.IOException: No space left on device
15/05/27 06:24:49 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000044_0, Status : FAILED
Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: = Could not find any valid local directory for = attempt_1432720271082_0005_m_000044_0_spill_0.out
at = org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathFor= Write(LocalDirAllocator.java:402)
at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:150)
at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:131)
at = org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFi= les.java:159)
at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java= :1584)
at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)<= br class=3D"">at = org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:720= )
at = org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at = org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native = Method)
at = javax.security.auth.Subject.doAs(Subject.java:415)
at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1628)
at = org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

15/05/27 06:24:50 INFO mapreduce.Job: Task Id = : attempt_1432720271082_0005_m_000045_0, Status : FAILED
Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: = Could not find any valid local directory for = attempt_1432720271082_0005_m_000045_0_spill_0.out
at = org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathFor= Write(LocalDirAllocator.java:402)
at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:150)
at = org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca= tor.java:131)
at = org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFi= les.java:159)
at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java= :1584)
at = org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)<= br class=3D"">at = org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:720= )
at = org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at = org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native = Method)
at = javax.security.auth.Subject.doAs(Subject.java:415)
at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1628)
at = org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

15/05/27 06:24:51 INFO mapreduce.Job: Task Id = : attempt_1432720271082_0005_m_000041_1, Status : FAILED
mkdir of = /hadoop/yarn/local/usercache/hdfs/appcache/application_1432720271082_0005/= container_1432720271082_0005_01_000050 failed

15/05/27 06:24:54 INFO mapreduce.Job: Task Id : = attempt_1432720271082_0005_m_000046_1, Status : FAILED
FSError: java.io.IOException: No space left on = device
 
Can = someone let me know how can I resolve this error ?
 
With = Regards,
Pratik Gadiya
 

DISCLAIMER =3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D This e-mail may contain privileged and confidential = information which is the property of Persistent Systems Ltd. It is = intended only for the use of the individual or entity to which it is = addressed. If you are not the intended recipient, you are not authorized = to read, retain, copy, print, distribute or use this message. If you = have received this communication in error, please notify the sender and = delete all copies of this message. Persistent Systems Ltd. does not = accept any liability for virus infected = mails.


= --Apple-Mail=_AA896689-93DA-4FE4-AE5C-EAF174C5CF9F--