Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <16027160.1154704034944.JavaMail.jira@brutus>
Date: Fri, 4 Aug 2006 08:07:14 -0700 (PDT)
From: =?utf-8?Q?Fr=C3=A9d=C3=A9ric_Bertin_=28JIRA=29?= <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Created: (HADOOP-424) mapreduce jobs fail when no split is
 returned via inputFormat.getSplits
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

mapreduce jobs fail when no split is returned via inputFormat.getSplits
-----------------------------------------------------------------------

                 Key: HADOOP-424
                 URL: http://issues.apache.org/jira/browse/HADOOP-424
             Project: Hadoop
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.4.0
            Reporter: Fr=C3=A9d=C3=A9ric Bertin


I'm using a MapReduce job to process some data logged and timestamped into =
files.
When the job runs, it does not process the whole data, but filters only the=
 data that has been logged since the last job run.

However, when no new data has been logged, the job fails because the getSpl=
its method of InputFormat returns no split. Thus the number of map tasks is=
 0. This is not intercepted, and the job fails at reduce step because it se=
ems it does not find any data to process:

java.io.FileNotFoundException: /local/home/hadoop/var/mapred/local/task_003=
0_r_000000_3/all.2 at org.apache.hadoop.fs.LocalFileSystem.openRaw(LocalFil=
eSystem.java:121) at org.apache.hadoop.fs.FSDataInputStream$Checker.(FSData=
InputStream.java:47) at org.apache.hadoop.fs.FSDataInputStream.(FSDataInput=
Stream.java:221) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:15=
0) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:259) at o=
rg.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:253) at org.apac=
he.hadoop.mapred.ReduceTask.run(ReduceTask.java:241) at org.apache.hadoop.m=
apred.TaskTracker$Child.main(TaskTracker.java:1013)

What should be Hadoop's behaviour in such a case?

IMHO, the job should be considered as successful. Indeed, this is not a job=
 failure, but just a lack of input data. WDYT?


--=20
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: htt=
p://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira