Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 096E911FE6 for ; Wed, 16 Jul 2014 17:03:07 +0000 (UTC) Received: (qmail 64263 invoked by uid 500); 16 Jul 2014 17:03:02 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 64152 invoked by uid 500); 16 Jul 2014 17:03:02 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 64142 invoked by uid 99); 16 Jul 2014 17:03:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 17:03:01 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [79.99.40.135] (HELO auth-smtp-02.streamline.net) (79.99.40.135) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jul 2014 17:02:55 +0000 Received: by auth-smtp-02.streamline.net (Postfix, from userid 500) id 1634317B2887; Wed, 16 Jul 2014 18:02:32 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on auth-smtp-02.streamline.net Received: from [192.168.0.4] (unknown [90.193.156.143]) (Authenticated sender: studio@chrismackenziephotography.co.uk) by auth-smtp-02.streamline.net (Postfix) with ESMTP id 3938917B2887 for ; Wed, 16 Jul 2014 18:02:30 +0100 (BST) User-Agent: Microsoft-MacOutlook/14.4.3.140616 Date: Wed, 16 Jul 2014 18:02:22 +0100 Subject: Can someone shed some light on this ? - java.io.IOException: Spill failed From: Chris MacKenzie To: Message-ID: Thread-Topic: Can someone shed some light on this ? - java.io.IOException: Spill failed Mime-version: 1.0 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi, Is this a coding or a setup issue ? I=B9m using Hadoop 2.41 My program is doing a concordance on 500,000 sequences of 400 chars. My cluster set is 32 data nodes and two masters. The exact error is: Error: java.io.IOException: Spill failed at=20 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas k.java:1535) at=20 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062) at=20 org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692) at=20 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput OutputContextImpl.java:89) at=20 org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp er.java:112) at=20 par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: 96) at=20 par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: 1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at=20 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1556) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1405523201400_0006_m_000000_0_spill_8.out at=20 org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW rite(LocalDirAllocator.java:402) at=20 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat or.java:150) at=20 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat or.java:131) at=20 org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil es.java:159) at=20 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java: 1566) at=20 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85 3) at=20 org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja va:1505) Regards, Chris