spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From marmb...@apache.org
Subject git commit: [SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location
Date Wed, 27 Aug 2014 22:06:07 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.1 b3d763b0b -> 77116875f


[SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location

Currently we do `relation.hiveQlTable.getDataLocation.getPath`, which returns the path-part
of the URI (e.g., "s3n://my-bucket/my-path" => "/my-path"). We should do `relation.hiveQlTable.getDataLocation.toString`
instead, as a URI's toString returns a faithful representation of the full URI, which can
later be passed into a Hadoop Path.

Author: Aaron Davidson <aaron@databricks.com>

Closes #2150 from aarondav/parquet-location and squashes the following commits:

459f72c [Aaron Davidson] [SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles
location

(cherry picked from commit cc275f4b7910f6d0ad266a43bac2fdae58e9739e)
Signed-off-by: Michael Armbrust <michael@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/77116875
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/77116875
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/77116875

Branch: refs/heads/branch-1.1
Commit: 77116875f4184e0a637d9d7fd5b1dfeaabe0c9d3
Parents: b3d763b
Author: Aaron Davidson <aaron@databricks.com>
Authored: Wed Aug 27 15:05:47 2014 -0700
Committer: Michael Armbrust <michael@databricks.com>
Committed: Wed Aug 27 15:06:04 2014 -0700

----------------------------------------------------------------------
 .../src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/77116875/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
index 10fa831..47e24f0 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
@@ -135,7 +135,7 @@ private[hive] trait HiveStrategies {
             .fakeOutput(projectList.map(_.toAttribute)):: Nil
         } else {
           hiveContext
-            .parquetFile(relation.hiveQlTable.getDataLocation.getPath)
+            .parquetFile(relation.hiveQlTable.getDataLocation.toString)
             .lowerCase
             .where(unresolvedOtherPredicates)
             .select(unresolvedProjection:_*)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message