hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkady Borkovsky <ark...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-619) Unify Map-Reduce and Streaming to take the same globbed input specification
Date Fri, 01 Dec 2006 17:25:38 GMT
+1 on Runping's comment.

It is very important to report the information about the matching input  
files to the user (as the output of Hadoop Streaming command)
-- each wildcard parameter that does not match any files, should be  
explicitly reported, but not to cause a failure
-- an exact (non-wildcard) parameter that does not match any file or  
directory has to be reported and CAUSE a failure
-- for every directory, the number of files in the directory should be  
-- if after successful wildcard matching the input has 0 files, the  
command should fail.

Failure implies
-- exit with non-0 return status
-- produce an error message explaining a failure.  No un-handled  

-- ab

On Dec 1, 2006, at 8:52 AM, Runping Qi (JIRA) wrote:

>     [  
> http://issues.apache.org/jira/browse/HADOOP-619? 
> page=comments#action_12454951 ]
> Runping Qi commented on HADOOP-619:
> -----------------------------------
> We should check the existence of the path(s), but leave the pattern  
> matching to InputFormatBase class.
> We have three cases to consider: all paths exist, non exists, and some  
> do, some do not.
> Cases 1 and 2 are easy. Case 3, I hink it is reasonable that the job  
> proceeds, but records that in the job
> status.
> When a job starts, we may should record all the input directories and  
> number of files under it matching the pattern. If no file matches the  
> pattern, the job should stop. Otherwise, the job generates splits from  
> the matched files and proceeds.
>> Unify Map-Reduce and Streaming to take the same globbed input  
>> specification
>> ---------------------------------------------------------------------- 
>> -----
>>                 Key: HADOOP-619
>>                 URL: http://issues.apache.org/jira/browse/HADOOP-619
>>             Project: Hadoop
>>          Issue Type: Improvement
>>          Components: mapred
>>            Reporter: eric baldeschwieler
>>         Assigned To: Sanjay Dahiya
>> Right now streaming input is specified very differently from other  
>> map-reduce input.  It would be good if these two apps could take much  
>> more similar input specs.
>> In particular -input in streaming expects a file or glob pattern  
>> while MR takes a directory.  It would be cool if both could take a  
>> glob patern of files and if both took a directory by default (with  
>> some patern excluded to allow logs, metadata and other framework  
>> output to be safely stored).
>> We want to be sure that MR input is backward compatible over this  
>> change.  I propose that a single file should be accepted as an input  
>> or a single directory.  Globs should only match directories if the  
>> paterns is '/' terminated, to avoid massive inputs specified by  
>> mistake.
>> Thoughts?
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:  
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:  
> http://www.atlassian.com/software/jira

View raw message