hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkady Borkovsky <ark...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-619) Unify Map-Reduce and Streaming to take the same globbed input specification
Date Mon, 04 Dec 2006 22:24:51 GMT
Is there a reason to split  "-input" and "-input-filter"?

If so, please make sure that there is a way to have
(a) different "-input-filter" for different "-input", and
(b) multiple "-input-filter" for the same "-input" (special case of (a))
(c) wild cards for specifying directories in "-input"



On Dec 4, 2006, at 2:17 PM, Owen O'Malley (JIRA) wrote:

>     [  
> http://issues.apache.org/jira/browse/HADOOP-619? 
> page=comments#action_12455431 ]
>
> Owen O'Malley commented on HADOOP-619:
> --------------------------------------
>
> I think that streaming should stop using the filename globbing for  
> --input and instead list a set of input directories.
>
> I think that the InputFormatBase should support a regex filter on  
> filenames that should default to something like "[^_].*" so that any  
> file that starts with an "_" is not treated as input. This will allow  
> us to put things like _LOGS as a filename to store the log files for a  
> job.
>
> Streaming should then support the regex filename filters using  
> "--input-filter" with a regex that filters the filenames.
>
>> Unify Map-Reduce and Streaming to take the same globbed input  
>> specification
>> ---------------------------------------------------------------------- 
>> -----
>>
>>                 Key: HADOOP-619
>>                 URL: http://issues.apache.org/jira/browse/HADOOP-619
>>             Project: Hadoop
>>          Issue Type: Improvement
>>          Components: mapred
>>            Reporter: eric baldeschwieler
>>         Assigned To: Sanjay Dahiya
>>
>> Right now streaming input is specified very differently from other  
>> map-reduce input.  It would be good if these two apps could take much  
>> more similar input specs.
>> In particular -input in streaming expects a file or glob pattern  
>> while MR takes a directory.  It would be cool if both could take a  
>> glob patern of files and if both took a directory by default (with  
>> some patern excluded to allow logs, metadata and other framework  
>> output to be safely stored).
>> We want to be sure that MR input is backward compatible over this  
>> change.  I propose that a single file should be accepted as an input  
>> or a single directory.  Globs should only match directories if the  
>> paterns is '/' terminated, to avoid massive inputs specified by  
>> mistake.
>> Thoughts?
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:  
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:  
> http://www.atlassian.com/software/jira
>
>


Mime
View raw message