Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=mime-version:in-reply-to:references:content-type:message-id:
	content-transfer-encoding:from:subject:date:to:x-mailer;
	b=IYUfKiKpVpKZg1OV852G+2znph6bZ+ETsVeaXItakiPmNWSJcxqyHrHFiDwSlrHQ
Mime-Version: 1.0 (Apple Message framework v752.3)
In-Reply-To: <608442.76296.qm@web53104.mail.re2.yahoo.com>
References: <608442.76296.qm@web53104.mail.re2.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <BB6598A9-4C4C-47D9-A44A-4C579EDAE54D@yahoo-inc.com>
Content-Transfer-Encoding: 7bit
From: "Owen O'Malley" <oom@yahoo-inc.com>
Subject: Re: A couple of usability problems
Date: Tue, 25 Sep 2007 23:00:57 -0700
To: hadoop-user@lucene.apache.org

On Sep 25, 2007, at 10:30 AM, Nathan Wang wrote:

> 1) Adjusting input set dynamically
> At the start, I had 9090 gzipped input data files for the job,
>     07/09/24 10:26:06 INFO mapred.FileInputFormat: Total input  
> paths to process : 9090
>
> Then I realized there were 3 files that were bad (couldn't be  
> gunzipped).
> So, I removed them by doing,
>     bin/hadoop  dfs  -rm  srcdir/FILExxx.gz
>
> 20 hours later, the job was failed.  And, I found a few errors in  
> the log:
>     org.apache.hadoop.ipc.RemoteException: java.io.IOException:  
> Cannot open filename ...FILExxx.gz
>
> Is it possible that the runtime could adjust the input data set  
> accordingly?

As Devaraj pointed out this is possible, but in general I think it is  
correct to make this an error. The planning for the job must happen  
at the beginning before the job is launched and once the map has been  
assigned a file, if the mapper can't read the assigned input, it is a  
fatal problem. If failures are tolerable for your application, you  
can set the percent of mappers and reducers that can fail before the  
job is killed.

> Can we check the existence of the output directory at the very  
> beginning, to save us a day?

It does already. That was done back before 0.1 in HADOOP-3. Was your  
program launching two jobs or something? Very strange.

-- Owen