Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <13164786.1165305626932.JavaMail.jira@brutus>
Date: Tue, 5 Dec 2006 00:00:26 -0800 (PST)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-619) Unify Map-Reduce and Streaming to
 take the same globbed input specification
In-Reply-To: <29129765.1161378094901.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/HADOOP-619?page=comments#action_12455537 ] 
            
Owen O'Malley commented on HADOOP-619:
--------------------------------------

just to clarify my proposal, it would look like:

for p in job.getInputPaths():
  for g in p.globPaths():
    if g.isFIle():
       fileList.add(g)
    else:
       for f in g.listFiles():
          fileList.add(f)

I don't understand what you are proposing, Eric. Is it:

for p in job.getInputPaths:
  if p.isDirectory():
     for f in p.listFiles():
        fileList.add(f)
  else:
     for g in p.globPaths():
         if g.isFile():
            fileList.add(g)

> Unify Map-Reduce and Streaming to take the same globbed input specification
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-619
>                 URL: http://issues.apache.org/jira/browse/HADOOP-619
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: eric baldeschwieler
>         Assigned To: Sanjay Dahiya
>
> Right now streaming input is specified very differently from other map-reduce input.  It would be good if these two apps could take much more similar input specs.
> In particular -input in streaming expects a file or glob pattern while MR takes a directory.  It would be cool if both could take a glob patern of files and if both took a directory by default (with some patern excluded to allow logs, metadata and other framework output to be safely stored).
> We want to be sure that MR input is backward compatible over this change.  I propose that a single file should be accepted as an input or a single directory.  Globs should only match directories if the paterns is '/' terminated, to avoid massive inputs specified by mistake.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira