spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HyukjinKwon <>
Subject [GitHub] spark pull request #15131: [SPARK-17577][SparkR] SparkR support add files to...
Date Sun, 18 Sep 2016 06:17:30 GMT
Github user HyukjinKwon commented on a diff in the pull request:
    --- Diff: R/pkg/R/context.R ---
    @@ -225,6 +225,37 @@ setCheckpointDir <- function(sc, dirName) {
       invisible(callJMethod(sc, "setCheckpointDir", suppressWarnings(normalizePath(dirName))))
    +#' Add a file or directory to be downloaded with this Spark job on every node.
    +#' The path passed can be either a local file, a file in HDFS (or other Hadoop-supported
    +#' filesystems), or an HTTP, HTTPS or FTP URI.  To access the file in Spark jobs,
    +#' use sparkFiles.get(fileName) to find its download location.
    +#' A directory can be given if the recursive option is set to true.
    +#' Currently directories are only supported for Hadoop-supported filesystems.
    +#' @param path The path of the files to be added
    +#' @param recursive Recursive or not if the path is directory. Default is FALSE.
    +#' @noRd
    +#' @examples
    +#' sc <- sparkR.init()
    +#' addFile(sc, "myfile")
    +addFile <- function(sc, path) {
    +  invisible(callJMethod(sc, "addFile", suppressWarnings(normalizePath(path))))
    --- End diff --
    Otherwise, I think we can not just `normalizePath` here to avoid this problem. I will
let you know after testing if that would really avoid this problem.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message