hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkady Borkovsky <ark...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-619) Unify Map-Reduce and Streaming to take the same globbed input specification
Date Tue, 05 Dec 2006 07:26:51 GMT
+1 for option 2.  Option 1 seems (a) incomplete, and (b) too complicated

(I understand option 2 as
-- there may be multiple inputs specified
-- each is a glob pattern matching one or more files or directories
-- if a pattern matches a directory, all files in the directory,  
excluding "hidden" files are include into the input.   That is  
)

On Dec 4, 2006, at 11:10 PM, Owen O'Malley (JIRA) wrote:

>     [  
> http://issues.apache.org/jira/browse/HADOOP-619? 
> page=comments#action_12455529 ]
>
> Owen O'Malley commented on HADOOP-619:
> --------------------------------------
>
> I really don't like this proposal. It adds custom syntax namly the ';'  
> operator that binds the wrong way in that it is tighter than ','. I'm  
> ok with either:
>
> 1. The input paths are all directories and there is a single glob or  
> regex filter on filenames.
> 2. The input paths are files or directories.
>
> In either case, I think that the input paths should be globbed by the  
> file system.
>
>> Unify Map-Reduce and Streaming to take the same globbed input  
>> specification
>> ---------------------------------------------------------------------- 
>> -----
>>
>>                 Key: HADOOP-619
>>                 URL: http://issues.apache.org/jira/browse/HADOOP-619
>>             Project: Hadoop
>>          Issue Type: Improvement
>>          Components: mapred
>>            Reporter: eric baldeschwieler
>>         Assigned To: Sanjay Dahiya
>>
>> Right now streaming input is specified very differently from other  
>> map-reduce input.  It would be good if these two apps could take much  
>> more similar input specs.
>> In particular -input in streaming expects a file or glob pattern  
>> while MR takes a directory.  It would be cool if both could take a  
>> glob patern of files and if both took a directory by default (with  
>> some patern excluded to allow logs, metadata and other framework  
>> output to be safely stored).
>> We want to be sure that MR input is backward compatible over this  
>> change.  I propose that a single file should be accepted as an input  
>> or a single directory.  Globs should only match directories if the  
>> paterns is '/' terminated, to avoid massive inputs specified by  
>> mistake.
>> Thoughts?
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:  
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:  
> http://www.atlassian.com/software/jira
>
>


Mime
View raw message