spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r...@apache.org
Subject spark git commit: [PySpark][Minor] Update sql example, so that can read file correctly
Date Sat, 25 Apr 2015 03:24:47 GMT
Repository: spark
Updated Branches:
  refs/heads/master 438859eb7 -> d874f8b54


[PySpark][Minor] Update sql example, so that can read file correctly

To run Spark, default will read file from HDFS if we don't set the schema.

Author: linweizhong <linweizhong@huawei.com>

Closes #5684 from Sephiroth-Lin/pyspark_example_minor and squashes the following commits:

19fe145 [linweizhong] Update example sql.py, so that can read file correctly


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d874f8b5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d874f8b5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d874f8b5

Branch: refs/heads/master
Commit: d874f8b546d8fae95bc92d8461b8189e51cb731b
Parents: 438859e
Author: linweizhong <linweizhong@huawei.com>
Authored: Fri Apr 24 20:23:19 2015 -0700
Committer: Reynold Xin <rxin@databricks.com>
Committed: Fri Apr 24 20:23:19 2015 -0700

----------------------------------------------------------------------
 examples/src/main/python/sql.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/d874f8b5/examples/src/main/python/sql.py
----------------------------------------------------------------------
diff --git a/examples/src/main/python/sql.py b/examples/src/main/python/sql.py
index 87d7b08..2c18875 100644
--- a/examples/src/main/python/sql.py
+++ b/examples/src/main/python/sql.py
@@ -18,6 +18,7 @@
 from __future__ import print_function
 
 import os
+import sys
 
 from pyspark import SparkContext
 from pyspark.sql import SQLContext
@@ -50,7 +51,11 @@ if __name__ == "__main__":
 
     # A JSON dataset is pointed to by path.
     # The path can be either a single text file or a directory storing text files.
-    path = os.path.join(os.environ['SPARK_HOME'], "examples/src/main/resources/people.json")
+    if len(sys.argv) < 2:
+        path = "file://" + \
+            os.path.join(os.environ['SPARK_HOME'], "examples/src/main/resources/people.json")
+    else:
+        path = sys.argv[1]
     # Create a DataFrame from the file(s) pointed to by path
     people = sqlContext.jsonFile(path)
     # root


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message