spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From l...@apache.org
Subject spark git commit: [SPARK-8458] [SQL] Don't strip scheme part of output path when writing ORC files
Date Fri, 19 Jun 2015 05:02:00 GMT
Repository: spark
Updated Branches:
  refs/heads/master 3eaed8769 -> a71cbbdea


[SPARK-8458] [SQL] Don't strip scheme part of output path when writing ORC files

`Path.toUri.getPath` strips scheme part of output path (from `file:///foo` to `/foo`), which
causes ORC data source only writes to the file system configured in Hadoop configuration.
Should use `Path.toString` instead.

Author: Cheng Lian <lian@databricks.com>

Closes #6892 from liancheng/spark-8458 and squashes the following commits:

87f8199 [Cheng Lian] Don't strip scheme of output path when writing ORC files


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a71cbbde
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a71cbbde
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a71cbbde

Branch: refs/heads/master
Commit: a71cbbdea581573192a59bf8472861c463c40fcb
Parents: 3eaed87
Author: Cheng Lian <lian@databricks.com>
Authored: Thu Jun 18 22:01:52 2015 -0700
Committer: Cheng Lian <lian@databricks.com>
Committed: Thu Jun 18 22:01:52 2015 -0700

----------------------------------------------------------------------
 .../src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/a71cbbde/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
index 77f1ca9..dbce39f 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
@@ -111,7 +111,7 @@ private[orc] class OrcOutputWriter(
     new OrcOutputFormat().getRecordWriter(
       new Path(path, filename).getFileSystem(conf),
       conf.asInstanceOf[JobConf],
-      new Path(path, filename).toUri.getPath,
+      new Path(path, filename).toString,
       Reporter.NULL
     ).asInstanceOf[RecordWriter[NullWritable, Writable]]
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message