hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-3162) Map/reduce stops working with comma separated input paths
Date Tue, 08 Apr 2008 05:15:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586318#action_12586318
] 

amareshwari edited comment on HADOOP-3162 at 4/7/08 10:14 PM:
--------------------------------------------------------------------------

This patch addresses the following:
1. Adds the following apis in FileInputFormat:
    * public static void setInputPaths(JobConf job, Path... paths);
    * public static void setInputPaths(JobConf job, String commaSeparatedPaths);
    * public static void addInputPath(JobConf job, Path path);
    * public static void addInputPaths(JobConf job, String commaSeparatedPaths);
2. Deprecates JobConf.setInputPath(Path) and JobConf.addInputPath(Path). And the methods have
0.16 semantics
3. Moves testInputPath() from conf/TestJobConf to mapred/TestInputPath. Added tests for testing
the new apis.


      was (Author: amareshwari):
    This patch addresses the following:
1. Adds the following apis in FileInputFormat:
    * public static void setInputPaths(JobConf job, Path... paths);
    * public static void setInputPaths(JobConf job, String commaSepatedPaths);
    * public static void addInputPath(JobConf job, Path path);
    * public static void addInputPaths(JobConf job, String commaSepatedPaths);
2. Deprecates JobConf.setInputPath(Path) and JobConf.addInputPath(Path)
3. Moves testInputPath() from conf/TestJobConf to mapred/TestInputPath. Added tests for testing
the new apis.

  
> Map/reduce stops working with comma separated input paths
> ---------------------------------------------------------
>
>                 Key: HADOOP-3162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3162
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Runping Qi
>            Assignee: Amareshwari Sriramadasu
>            Priority: Blocker
>             Fix For: 0.17.0
>
>         Attachments: patch-3162.txt, patch-3162.txt
>
>
> When a job is given a comma separated input file list, FileInputFormat class throws an
exception, complaining the input is invalid:
> org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
> namenode:port/gridmix/data/MonsterQueryBlockCompressed/part-
> 00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
> QueryBlockCompressed/part-00002
>         at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
> t.java:213)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
>         at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
> rator.java:189)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message