spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Haviv (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-7155) SparkContext's newAPIHadoopFile does not support comma-separated list of files, but the other API hadoopFile does.
Date Tue, 18 Aug 2015 12:23:46 GMT

    [ https://issues.apache.org/jira/browse/SPARK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701176#comment-14701176
] 

Daniel Haviv commented on SPARK-7155:
-------------------------------------

Hi,
It doesn't seem to work. When I'm trying to pass binaryFiles a comma seperated list of paths
I'm getting the following exception:
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://UBUNTU1:8020/user/administrator/pictures/CASIA-WebFace-after-filtering/1026045


Thank you.
Daniel

> SparkContext's newAPIHadoopFile does not support comma-separated list of files, but the
other API hadoopFile does.
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-7155
>                 URL: https://issues.apache.org/jira/browse/SPARK-7155
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.3.1
>         Environment: Ubuntu 14.04
>            Reporter: Yong Tang
>            Assignee: Yong Tang
>             Fix For: 1.3.2, 1.4.0
>
>
> SparkContext's newAPIHadoopFile() does not support comma-separated list of files. For
example, the following:
> sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt", classOf[TextInputFormat], classOf[LongWritable],
classOf[Text])
> will throw
> org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist:
file:/root/file1.txt,/root/file2.txt
> However, the other API hadoopFile() is able to process comma-separated list of files
correctly.
> In addition, since sc.textFile() uses hadoopFile(), it is also able to process comma-separated
list of files correctly.
> The problem is that newAPIHadoopFile() use addInputPath() to add the file path into NewHadoopRDD.
See Ln 928-931, master branch:
>     val job = new NewHadoopJob(conf)
>     NewFileInputFormat.addInputPath(job, new Path(path))
>     val updatedConf = job.getConfiguration
>     new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path)
> Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will resolve this
issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message