Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 39656 invoked from network); 23 Aug 2006 07:48:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 23 Aug 2006 07:48:28 -0000 Received: (qmail 62950 invoked by uid 500); 23 Aug 2006 07:48:23 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 62768 invoked by uid 500); 23 Aug 2006 07:48:23 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 62759 invoked by uid 99); 23 Aug 2006 07:48:22 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Aug 2006 00:48:22 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [209.237.227.198] (HELO brutus.apache.org) (209.237.227.198) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Aug 2006 00:48:22 -0700 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id E490A410026 for ; Wed, 23 Aug 2006 07:45:15 +0000 (GMT) Message-ID: <8004387.1156319115934.JavaMail.jira@brutus> Date: Wed, 23 Aug 2006 00:45:15 -0700 (PDT) From: =?utf-8?Q?Fr=C3=A9d=C3=A9ric_Bertin_=28JIRA=29?= To: hadoop-dev@lucene.apache.org Subject: [jira] Updated: (HADOOP-424) mapreduce jobs fail when no split is returned via inputFormat.getSplits In-Reply-To: <16027160.1154704034944.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-424?page=3Dall ] Fr=C3=A9d=C3=A9ric Bertin updated HADOOP-424: ----------------------------------- Attachment: emptyJobTest.patch > mapreduce jobs fail when no split is returned via inputFormat.getSplits > ----------------------------------------------------------------------- > > Key: HADOOP-424 > URL: http://issues.apache.org/jira/browse/HADOOP-424 > Project: Hadoop > Issue Type: Bug > Components: mapred > Affects Versions: 0.5.0 > Reporter: Fr=C3=A9d=C3=A9ric Bertin > Attachments: emptyJobTest.patch > > > I'm using a MapReduce job to process some data logged and timestamped int= o files. > When the job runs, it does not process the whole data, but filters only t= he data that has been logged since the last job run. > However, when no new data has been logged, the job fails because the getS= plits method of InputFormat returns no split. Thus the number of map tasks = is 0. This is not intercepted, and the job fails at reduce step because it = seems it does not find any data to process: > java.io.FileNotFoundException: /local/home/hadoop/var/mapred/local/task_0= 030_r_000000_3/all.2 at org.apache.hadoop.fs.LocalFileSystem.openRaw(LocalF= ileSystem.java:121) at org.apache.hadoop.fs.FSDataInputStream$Checker.(FSDa= taInputStream.java:47) at org.apache.hadoop.fs.FSDataInputStream.(FSDataInp= utStream.java:221) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:= 150) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:259) at= org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:253) at org.ap= ache.hadoop.mapred.ReduceTask.run(ReduceTask.java:241) at org.apache.hadoop= .mapred.TaskTracker$Child.main(TaskTracker.java:1013) > What should be Hadoop's behaviour in such a case? > IMHO, the job should be considered as successful. Indeed, this is not a j= ob failure, but just a lack of input data. WDYT? --=20 This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: htt= p://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira