spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-8365) pyspark does not retain --packages or --jars passed on the command line as of 1.4.0
Date Wed, 17 Jun 2015 17:41:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or updated SPARK-8365:
-----------------------------
    Priority: Blocker  (was: Major)

> pyspark does not retain --packages or --jars passed on the command line as of 1.4.0
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-8365
>                 URL: https://issues.apache.org/jira/browse/SPARK-8365
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 1.4.0
>            Reporter: Don Drake
>            Priority: Blocker
>
> I downloaded the pre-compiled Spark 1.4.0 and attempted to run an existing Python Spark
application against it and got the following error:
> py4j.protocol.Py4JJavaError: An error occurred while calling o90.save.
> : java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.csv
> I pass the following on the command-line to my spark-submit:
> --packages com.databricks:spark-csv_2.10:1.0.3
> This worked fine on 1.3.1, but not in 1.4.
> I was able to replicate it with the following pyspark:
> {code}
> a = {'a':1.0, 'b':'asdf'}
> rdd = sc.parallelize([a])
> df = sqlContext.createDataFrame(rdd)
> df.save("/tmp/d.csv", "com.databricks.spark.csv")
> {code}
> Even using the new df.write.format('com.databricks.spark.csv').save('/tmp/d.csv') gives
the same error. 
> I see it was added in the web UI:
> file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar	Added By User
> file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar	Added By User
> http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jar	Added By User
> http://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jar	Added By User
> Thoughts?
> *I also attempted using the Scala spark-shell to load a csv using the same package and
it worked just fine, so this seems specific to pyspark.*
> -Don
> Gory details:
> {code}
> $ pyspark --packages "com.databricks:spark-csv_2.10:1.0.3"
> Python 2.7.6 (default, Sep  9 2014, 15:04:36)
> [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> Ivy Default Cache set to: /Users/drake/.ivy2/cache
> The jars for the packages stored in: /Users/drake/.ivy2/jars
> :: loading settings :: url = jar:file:/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/lib/spark-assembly-1.4.0-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> com.databricks#spark-csv_2.10 added as a dependency
> :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
> 	confs: [default]
> 	found com.databricks#spark-csv_2.10;1.0.3 in central
> 	found org.apache.commons#commons-csv;1.1 in central
> :: resolution report :: resolve 590ms :: artifacts dl 17ms
> 	:: modules in use:
> 	com.databricks#spark-csv_2.10;1.0.3 from central in [default]
> 	org.apache.commons#commons-csv;1.1 from central in [default]
> 	---------------------------------------------------------------------
> 	|                  |            modules            ||   artifacts   |
> 	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
> 	---------------------------------------------------------------------
> 	|      default     |   2   |   0   |   0   |   0   ||   2   |   0   |
> 	---------------------------------------------------------------------
> :: retrieving :: org.apache.spark#spark-submit-parent
> 	confs: [default]
> 	0 artifacts copied, 2 already retrieved (0kB/15ms)
> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> 15/06/13 11:06:08 INFO SparkContext: Running Spark version 1.4.0
> 2015-06-13 11:06:08.921 java[19233:2145789] Unable to load realm info from SCDynamicStore
> 15/06/13 11:06:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
> 15/06/13 11:06:09 WARN Utils: Your hostname, Dons-MacBook-Pro-2.local resolves to a loopback
address: 127.0.0.1; using 10.0.0.222 instead (on interface en0)
> 15/06/13 11:06:09 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
> 15/06/13 11:06:09 INFO SecurityManager: Changing view acls to: drake
> 15/06/13 11:06:09 INFO SecurityManager: Changing modify acls to: drake
> 15/06/13 11:06:09 INFO SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(drake); users with modify permissions: Set(drake)
> 15/06/13 11:06:10 INFO Slf4jLogger: Slf4jLogger started
> 15/06/13 11:06:10 INFO Remoting: Starting remoting
> 15/06/13 11:06:10 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.0.0.222:56870]
> 15/06/13 11:06:10 INFO Utils: Successfully started service 'sparkDriver' on port 56870.
> 15/06/13 11:06:10 INFO SparkEnv: Registering MapOutputTracker
> 15/06/13 11:06:10 INFO SparkEnv: Registering BlockManagerMaster
> 15/06/13 11:06:10 INFO DiskBlockManager: Created local directory at /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/blockmgr-a1412b71-fe56-429c-a193-ce3fb95d2ffd
> 15/06/13 11:06:10 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
> 15/06/13 11:06:10 INFO HttpFileServer: HTTP File server directory is /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/httpd-84d178da-7e60-4eed-8031-e6a0c465bd4c
> 15/06/13 11:06:10 INFO HttpServer: Starting HTTP Server
> 15/06/13 11:06:10 INFO Utils: Successfully started service 'HTTP file server' on port
56871.
> 15/06/13 11:06:10 INFO SparkEnv: Registering OutputCommitCoordinator
> 15/06/13 11:06:11 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting
port 4041.
> 15/06/13 11:06:11 INFO Utils: Successfully started service 'SparkUI' on port 4041.
> 15/06/13 11:06:11 INFO SparkUI: Started SparkUI at http://10.0.0.222:4041
> 15/06/13 11:06:11 INFO SparkContext: Added JAR file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar
at http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jar with timestamp 1434211571303
> 15/06/13 11:06:11 INFO SparkContext: Added JAR file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar
at http://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jar with timestamp 1434211571326
> 15/06/13 11:06:11 INFO Utils: Copying /Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar
to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar
> 15/06/13 11:06:11 INFO SparkContext: Added file file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar
at file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar with timestamp 1434211571468
> 15/06/13 11:06:11 INFO Utils: Copying /Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar
to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar
> 15/06/13 11:06:11 INFO SparkContext: Added file file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar
at file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar with timestamp 1434211571502
> 15/06/13 11:06:11 INFO Executor: Starting executor ID driver on host localhost
> 15/06/13 11:06:11 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService'
on port 56872.
> 15/06/13 11:06:11 INFO NettyBlockTransferService: Server created on 56872
> 15/06/13 11:06:11 INFO BlockManagerMaster: Trying to register BlockManager
> 15/06/13 11:06:11 INFO BlockManagerMasterEndpoint: Registering block manager localhost:56872
with 265.4 MB RAM, BlockManagerId(driver, localhost, 56872)
> 15/06/13 11:06:11 INFO BlockManagerMaster: Registered BlockManager
> Welcome to
>       ____              __
>      / __/__  ___ _____/ /__
>     _\ \/ _ \/ _ `/ __/  '_/
>    /__ / .__/\_,_/_/ /_/\_\   version 1.4.0
>       /_/
> Using Python version 2.7.6 (default, Sep  9 2014 15:04:36)
> SparkContext available as sc, HiveContext available as sqlContext.
> >>> a = {'a':1.0, 'b':'asdf'}
> >>> rdd = sc.parallelize([a])
> >>> df = sqlContext.createDataFrame(rdd)
> 15/06/13 11:06:50 INFO SparkContext: Starting job: runJob at PythonRDD.scala:366
> 15/06/13 11:06:50 INFO DAGScheduler: Got job 0 (runJob at PythonRDD.scala:366) with 1
output partitions (allowLocal=true)
> 15/06/13 11:06:50 INFO DAGScheduler: Final stage: ResultStage 0(runJob at PythonRDD.scala:366)
> 15/06/13 11:06:50 INFO DAGScheduler: Parents of final stage: List()
> 15/06/13 11:06:50 INFO DAGScheduler: Missing parents: List()
> 15/06/13 11:06:50 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[1] at RDD at
PythonRDD.scala:43), which has no missing parents
> 15/06/13 11:06:51 INFO MemoryStore: ensureFreeSpace(3672) called with curMem=0, maxMem=278302556
> 15/06/13 11:06:51 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated
size 3.6 KB, free 265.4 MB)
> 15/06/13 11:06:51 INFO MemoryStore: ensureFreeSpace(2328) called with curMem=3672, maxMem=278302556
> 15/06/13 11:06:51 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory
(estimated size 2.3 KB, free 265.4 MB)
> 15/06/13 11:06:51 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:56872
(size: 2.3 KB, free: 265.4 MB)
> 15/06/13 11:06:51 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:874
> 15/06/13 11:06:51 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (PythonRDD[1]
at RDD at PythonRDD.scala:43)
> 15/06/13 11:06:51 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
> 15/06/13 11:06:51 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:51 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 15/06/13 11:06:51 INFO Executor: Fetching file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar
with timestamp 1434211571502
> 15/06/13 11:06:51 INFO Utils: /Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar
has been previously copied to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar
> 15/06/13 11:06:51 INFO Executor: Fetching file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar
with timestamp 1434211571468
> 15/06/13 11:06:51 INFO Utils: /Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar
has been previously copied to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar
> 15/06/13 11:06:51 INFO Executor: Fetching http://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jar
with timestamp 1434211571326
> 15/06/13 11:06:51 INFO Utils: Fetching http://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jar
to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp2449082240048543653.tmp
> 15/06/13 11:06:51 INFO Utils: /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp2449082240048543653.tmp
has been previously copied to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar
> 15/06/13 11:06:51 INFO Executor: Adding file:/private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar
to class loader
> 15/06/13 11:06:51 INFO Executor: Fetching http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jar
with timestamp 1434211571303
> 15/06/13 11:06:51 INFO Utils: Fetching http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jar
to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp1396931258018379545.tmp
> 15/06/13 11:06:51 INFO Utils: /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp1396931258018379545.tmp
has been previously copied to /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar
> 15/06/13 11:06:51 INFO Executor: Adding file:/private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar
to class loader
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 3165, boot = 3155, init = 10, finish
= 0
> 15/06/13 11:06:54 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 666 bytes result
sent to driver
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3505
ms on localhost (1/1)
> 15/06/13 11:06:54 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed,
from pool
> 15/06/13 11:06:54 INFO DAGScheduler: ResultStage 0 (runJob at PythonRDD.scala:366) finished
in 3.525 s
> 15/06/13 11:06:54 INFO DAGScheduler: Job 0 finished: runJob at PythonRDD.scala:366, took
3.852112 s
> 15/06/13 11:06:54 INFO SparkContext: Starting job: runJob at PythonRDD.scala:366
> 15/06/13 11:06:54 INFO DAGScheduler: Got job 1 (runJob at PythonRDD.scala:366) with 4
output partitions (allowLocal=true)
> 15/06/13 11:06:54 INFO DAGScheduler: Final stage: ResultStage 1(runJob at PythonRDD.scala:366)
> 15/06/13 11:06:54 INFO DAGScheduler: Parents of final stage: List()
> 15/06/13 11:06:54 INFO DAGScheduler: Missing parents: List()
> 15/06/13 11:06:54 INFO DAGScheduler: Submitting ResultStage 1 (PythonRDD[2] at RDD at
PythonRDD.scala:43), which has no missing parents
> 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(3672) called with curMem=6000, maxMem=278302556
> 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated
size 3.6 KB, free 265.4 MB)
> 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(2330) called with curMem=9672, maxMem=278302556
> 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory
(estimated size 2.3 KB, free 265.4 MB)
> 15/06/13 11:06:54 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:56872
(size: 2.3 KB, free: 265.4 MB)
> 15/06/13 11:06:54 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874
> 15/06/13 11:06:54 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (PythonRDD[2]
at RDD at PythonRDD.scala:43)
> 15/06/13 11:06:54 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:54 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
> 15/06/13 11:06:54 INFO Executor: Running task 1.0 in stage 1.0 (TID 2)
> 15/06/13 11:06:54 INFO Executor: Running task 2.0 in stage 1.0 (TID 3)
> 15/06/13 11:06:54 INFO Executor: Running task 3.0 in stage 1.0 (TID 4)
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -15, init = 17, finish = 0
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 9, boot = 6, init = 2, finish = 1
> 15/06/13 11:06:54 INFO Executor: Finished task 3.0 in stage 1.0 (TID 4). 666 bytes result
sent to driver
> 15/06/13 11:06:54 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 666 bytes result
sent to driver
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 13, boot = 9, init = 4, finish = 0
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 24 ms
on localhost (1/4)
> 15/06/13 11:06:54 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). 666 bytes result
sent to driver
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 28 ms
on localhost (2/4)
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 27 ms
on localhost (3/4)
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 28, boot = 28, init = 0, finish = 0
> 15/06/13 11:06:54 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 666 bytes result
sent to driver
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 42 ms
on localhost (4/4)
> 15/06/13 11:06:54 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed,
from pool
> 15/06/13 11:06:54 INFO DAGScheduler: ResultStage 1 (runJob at PythonRDD.scala:366) finished
in 0.044 s
> 15/06/13 11:06:54 INFO DAGScheduler: Job 1 finished: runJob at PythonRDD.scala:366, took
0.063304 s
> 15/06/13 11:06:54 INFO SparkContext: Starting job: runJob at PythonRDD.scala:366
> 15/06/13 11:06:54 INFO DAGScheduler: Got job 2 (runJob at PythonRDD.scala:366) with 3
output partitions (allowLocal=true)
> 15/06/13 11:06:54 INFO DAGScheduler: Final stage: ResultStage 2(runJob at PythonRDD.scala:366)
> 15/06/13 11:06:54 INFO DAGScheduler: Parents of final stage: List()
> 15/06/13 11:06:54 INFO DAGScheduler: Missing parents: List()
> 15/06/13 11:06:54 INFO DAGScheduler: Submitting ResultStage 2 (PythonRDD[3] at RDD at
PythonRDD.scala:43), which has no missing parents
> 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(3672) called with curMem=12002, maxMem=278302556
> 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated
size 3.6 KB, free 265.4 MB)
> 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(2330) called with curMem=15674, maxMem=278302556
> 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory
(estimated size 2.3 KB, free 265.4 MB)
> 15/06/13 11:06:54 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:56872
(size: 2.3 KB, free: 265.4 MB)
> 15/06/13 11:06:54 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874
> 15/06/13 11:06:54 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 2 (PythonRDD[3]
at RDD at PythonRDD.scala:43)
> 15/06/13 11:06:54 INFO TaskSchedulerImpl: Adding task set 2.0 with 3 tasks
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 5, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 6, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 7, localhost,
PROCESS_LOCAL, 1708 bytes)
> 15/06/13 11:06:54 INFO Executor: Running task 0.0 in stage 2.0 (TID 5)
> 15/06/13 11:06:54 INFO Executor: Running task 1.0 in stage 2.0 (TID 6)
> 15/06/13 11:06:54 INFO Executor: Running task 2.0 in stage 2.0 (TID 7)
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -41, init = 43, finish = 0
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -38, init = 40, finish = 0
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 1, boot = -77, init = 78, finish = 0
> 15/06/13 11:06:54 INFO Executor: Finished task 0.0 in stage 2.0 (TID 5). 666 bytes result
sent to driver
> 15/06/13 11:06:54 INFO Executor: Finished task 1.0 in stage 2.0 (TID 6). 666 bytes result
sent to driver
> 15/06/13 11:06:54 INFO Executor: Finished task 2.0 in stage 2.0 (TID 7). 722 bytes result
sent to driver
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 5) in 14 ms
on localhost (1/3)
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 6) in 13 ms
on localhost (2/3)
> 15/06/13 11:06:54 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 7) in 13 ms
on localhost (3/3)
> 15/06/13 11:06:54 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed,
from pool
> 15/06/13 11:06:54 INFO DAGScheduler: ResultStage 2 (runJob at PythonRDD.scala:366) finished
in 0.016 s
> 15/06/13 11:06:54 INFO DAGScheduler: Job 2 finished: runJob at PythonRDD.scala:366, took
0.087522 s
> /Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/context.py:198: UserWarning:
Using RDD of dict to inferSchema is deprecated,please use pyspark.sql.Row instead
>   warnings.warn("Using RDD of dict to inferSchema is deprecated,"
> 15/06/13 11:06:54 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:56872
in memory (size: 2.3 KB, free: 265.4 MB)
> 15/06/13 11:06:54 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:56872
in memory (size: 2.3 KB, free: 265.4 MB)
> 15/06/13 11:06:54 INFO SparkContext: Starting job: runJob at PythonRDD.scala:366
> 15/06/13 11:06:54 INFO DAGScheduler: Got job 3 (runJob at PythonRDD.scala:366) with 1
output partitions (allowLocal=true)
> 15/06/13 11:06:54 INFO DAGScheduler: Final stage: ResultStage 3(runJob at PythonRDD.scala:366)
> 15/06/13 11:06:54 INFO DAGScheduler: Parents of final stage: List()
> 15/06/13 11:06:54 INFO DAGScheduler: Missing parents: List()
> 15/06/13 11:06:54 INFO DAGScheduler: Submitting ResultStage 3 (PythonRDD[4] at RDD at
PythonRDD.scala:43), which has no missing parents
> 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(5120) called with curMem=6002, maxMem=278302556
> 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated
size 5.0 KB, free 265.4 MB)
> 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(3338) called with curMem=11122, maxMem=278302556
> 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory
(estimated size 3.3 KB, free 265.4 MB)
> 15/06/13 11:06:54 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:56872
(size: 3.3 KB, free: 265.4 MB)
> 15/06/13 11:06:54 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:874
> 15/06/13 11:06:54 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (PythonRDD[4]
at RDD at PythonRDD.scala:43)
> 15/06/13 11:06:54 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
> 15/06/13 11:06:54 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 8, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:54 INFO Executor: Running task 0.0 in stage 3.0 (TID 8)
> 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -29, init = 31, finish = 0
> 15/06/13 11:06:55 INFO Executor: Finished task 0.0 in stage 3.0 (TID 8). 666 bytes result
sent to driver
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 8) in 10 ms
on localhost (1/1)
> 15/06/13 11:06:55 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed,
from pool
> 15/06/13 11:06:55 INFO DAGScheduler: ResultStage 3 (runJob at PythonRDD.scala:366) finished
in 0.011 s
> 15/06/13 11:06:55 INFO DAGScheduler: Job 3 finished: runJob at PythonRDD.scala:366, took
0.035088 s
> 15/06/13 11:06:55 INFO SparkContext: Starting job: runJob at PythonRDD.scala:366
> 15/06/13 11:06:55 INFO DAGScheduler: Got job 4 (runJob at PythonRDD.scala:366) with 4
output partitions (allowLocal=true)
> 15/06/13 11:06:55 INFO DAGScheduler: Final stage: ResultStage 4(runJob at PythonRDD.scala:366)
> 15/06/13 11:06:55 INFO DAGScheduler: Parents of final stage: List()
> 15/06/13 11:06:55 INFO DAGScheduler: Missing parents: List()
> 15/06/13 11:06:55 INFO DAGScheduler: Submitting ResultStage 4 (PythonRDD[5] at RDD at
PythonRDD.scala:43), which has no missing parents
> 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(5120) called with curMem=14460, maxMem=278302556
> 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated
size 5.0 KB, free 265.4 MB)
> 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(3337) called with curMem=19580, maxMem=278302556
> 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory
(estimated size 3.3 KB, free 265.4 MB)
> 15/06/13 11:06:55 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:56872
(size: 3.3 KB, free: 265.4 MB)
> 15/06/13 11:06:55 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:874
> 15/06/13 11:06:55 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 4 (PythonRDD[5]
at RDD at PythonRDD.scala:43)
> 15/06/13 11:06:55 INFO TaskSchedulerImpl: Adding task set 4.0 with 4 tasks
> 15/06/13 11:06:55 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 9, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:55 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 10, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:55 INFO TaskSetManager: Starting task 2.0 in stage 4.0 (TID 11, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:55 INFO TaskSetManager: Starting task 3.0 in stage 4.0 (TID 12, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:55 INFO Executor: Running task 3.0 in stage 4.0 (TID 12)
> 15/06/13 11:06:55 INFO Executor: Running task 1.0 in stage 4.0 (TID 10)
> 15/06/13 11:06:55 INFO Executor: Running task 2.0 in stage 4.0 (TID 11)
> 15/06/13 11:06:55 INFO Executor: Running task 0.0 in stage 4.0 (TID 9)
> 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -79, init = 80, finish = 0
> 15/06/13 11:06:55 INFO Executor: Finished task 3.0 in stage 4.0 (TID 12). 666 bytes result
sent to driver
> 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -27, init = 28, finish = 0
> 15/06/13 11:06:55 INFO Executor: Finished task 1.0 in stage 4.0 (TID 10). 666 bytes result
sent to driver
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 3.0 in stage 4.0 (TID 12) in 11
ms on localhost (1/4)
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 10) in 14
ms on localhost (2/4)
> 15/06/13 11:06:55 INFO PythonRDD: Times: total = 22, boot = 22, init = 0, finish = 0
> 15/06/13 11:06:55 INFO PythonRDD: Times: total = 21, boot = 21, init = 0, finish = 0
> 15/06/13 11:06:55 INFO Executor: Finished task 2.0 in stage 4.0 (TID 11). 666 bytes result
sent to driver
> 15/06/13 11:06:55 INFO Executor: Finished task 0.0 in stage 4.0 (TID 9). 666 bytes result
sent to driver
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 2.0 in stage 4.0 (TID 11) in 37
ms on localhost (3/4)
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 9) in 43 ms
on localhost (4/4)
> 15/06/13 11:06:55 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed,
from pool
> 15/06/13 11:06:55 INFO DAGScheduler: ResultStage 4 (runJob at PythonRDD.scala:366) finished
in 0.044 s
> 15/06/13 11:06:55 INFO DAGScheduler: Job 4 finished: runJob at PythonRDD.scala:366, took
0.059163 s
> 15/06/13 11:06:55 INFO SparkContext: Starting job: runJob at PythonRDD.scala:366
> 15/06/13 11:06:55 INFO DAGScheduler: Got job 5 (runJob at PythonRDD.scala:366) with 3
output partitions (allowLocal=true)
> 15/06/13 11:06:55 INFO DAGScheduler: Final stage: ResultStage 5(runJob at PythonRDD.scala:366)
> 15/06/13 11:06:55 INFO DAGScheduler: Parents of final stage: List()
> 15/06/13 11:06:55 INFO DAGScheduler: Missing parents: List()
> 15/06/13 11:06:55 INFO DAGScheduler: Submitting ResultStage 5 (PythonRDD[6] at RDD at
PythonRDD.scala:43), which has no missing parents
> 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(5120) called with curMem=22917, maxMem=278302556
> 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated
size 5.0 KB, free 265.4 MB)
> 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(3338) called with curMem=28037, maxMem=278302556
> 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory
(estimated size 3.3 KB, free 265.4 MB)
> 15/06/13 11:06:55 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:56872
(size: 3.3 KB, free: 265.4 MB)
> 15/06/13 11:06:55 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:874
> 15/06/13 11:06:55 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 5 (PythonRDD[6]
at RDD at PythonRDD.scala:43)
> 15/06/13 11:06:55 INFO TaskSchedulerImpl: Adding task set 5.0 with 3 tasks
> 15/06/13 11:06:55 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 13, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:55 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 14, localhost,
PROCESS_LOCAL, 1665 bytes)
> 15/06/13 11:06:55 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 15, localhost,
PROCESS_LOCAL, 1708 bytes)
> 15/06/13 11:06:55 INFO Executor: Running task 0.0 in stage 5.0 (TID 13)
> 15/06/13 11:06:55 INFO Executor: Running task 1.0 in stage 5.0 (TID 14)
> 15/06/13 11:06:55 INFO Executor: Running task 2.0 in stage 5.0 (TID 15)
> 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -24, init = 25, finish = 0
> 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -24, init = 25, finish = 0
> 15/06/13 11:06:55 INFO Executor: Finished task 0.0 in stage 5.0 (TID 13). 666 bytes result
sent to driver
> 15/06/13 11:06:55 INFO Executor: Finished task 2.0 in stage 5.0 (TID 15). 716 bytes result
sent to driver
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 13) in 12
ms on localhost (1/3)
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID 15) in 11
ms on localhost (2/3)
> 15/06/13 11:06:55 INFO PythonRDD: Times: total = 21, boot = 20, init = 1, finish = 0
> 15/06/13 11:06:55 INFO Executor: Finished task 1.0 in stage 5.0 (TID 14). 666 bytes result
sent to driver
> 15/06/13 11:06:55 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 14) in 36
ms on localhost (3/3)
> 15/06/13 11:06:55 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed,
from pool
> 15/06/13 11:06:55 INFO DAGScheduler: ResultStage 5 (runJob at PythonRDD.scala:366) finished
in 0.038 s
> 15/06/13 11:06:55 INFO DAGScheduler: Job 5 finished: runJob at PythonRDD.scala:366, took
0.052978 s
> 15/06/13 11:06:56 INFO HiveContext: Initializing execution hive, version 0.13.1
> 15/06/13 11:06:56 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 15/06/13 11:06:56 INFO ObjectStore: ObjectStore, initialize called
> 15/06/13 11:06:57 INFO Persistence: Property datanucleus.cache.level2 unknown - will
be ignored
> 15/06/13 11:06:57 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown
- will be ignored
> 15/06/13 11:06:57 WARN Connection: BoneCP specified but not present in CLASSPATH (or
one of dependencies)
> 15/06/13 11:06:58 WARN Connection: BoneCP specified but not present in CLASSPATH (or
one of dependencies)
> 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_5_piece0 on localhost:56872
in memory (size: 3.3 KB, free: 265.4 MB)
> 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:56872
in memory (size: 3.3 KB, free: 265.4 MB)
> 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:56872
in memory (size: 3.3 KB, free: 265.4 MB)
> 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_2_piece0 on localhost:56872
in memory (size: 2.3 KB, free: 265.4 MB)
> 15/06/13 11:07:00 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 15/06/13 11:07:00 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on
mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".
> 15/06/13 11:07:01 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:01 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:04 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:04 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:04 INFO ObjectStore: Initialized ObjectStore
> 15/06/13 11:07:04 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification
is not enabled so recording the schema version 0.13.1aa
> 15/06/13 11:07:05 INFO HiveMetaStore: Added admin role in metastore
> 15/06/13 11:07:05 INFO HiveMetaStore: Added public role in metastore
> 15/06/13 11:07:05 INFO HiveMetaStore: No user is added in admin role, since config is
empty
> 15/06/13 11:07:05 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
> 15/06/13 11:07:06 INFO HiveContext: Initializing HiveMetastoreConnection version 0.13.1
using Spark classes.
> 15/06/13 11:07:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
> 15/06/13 11:07:08 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 15/06/13 11:07:08 INFO ObjectStore: ObjectStore, initialize called
> 15/06/13 11:07:08 INFO Persistence: Property datanucleus.cache.level2 unknown - will
be ignored
> 15/06/13 11:07:08 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown
- will be ignored
> 15/06/13 11:07:08 WARN Connection: BoneCP specified but not present in CLASSPATH (or
one of dependencies)
> 15/06/13 11:07:09 WARN Connection: BoneCP specified but not present in CLASSPATH (or
one of dependencies)
> 15/06/13 11:07:11 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 15/06/13 11:07:11 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on
mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".
> 15/06/13 11:07:12 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:12 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:13 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:13 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
> 15/06/13 11:07:14 INFO ObjectStore: Initialized ObjectStore
> 15/06/13 11:07:14 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification
is not enabled so recording the schema version 0.13.1aa
> 15/06/13 11:07:14 INFO HiveMetaStore: Added admin role in metastore
> 15/06/13 11:07:15 INFO HiveMetaStore: Added public role in metastore
> 15/06/13 11:07:15 INFO HiveMetaStore: No user is added in admin role, since config is
empty
> 15/06/13 11:07:15 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
> >>>
> >>> df.save("/tmp/d.csv", "com.databricks.spark.csv")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/dataframe.py",
line 202, in save
>     return self.write.save(path, source, mode, **options)
>   File "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/readwriter.py",
line 295, in save
>     self._jwrite.save(path)
>   File "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
line 538, in __call__
>   File "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py",
line 300, in get_return_value
> py4j.protocol.Py4JJavaError: An error occurred while calling o86.save.
> : java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.csv
> 	at scala.sys.package$.error(package.scala:27)
> 	at org.apache.spark.sql.sources.ResolvedDataSource$.lookupDataSource(ddl.scala:216)
> 	at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:302)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:144)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:135)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
> 	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
> 	at py4j.Gateway.invoke(Gateway.java:259)
> 	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
> 	at py4j.commands.CallCommand.execute(CallCommand.java:79)
> 	at py4j.GatewayConnection.run(GatewayConnection.java:207)
> 	at java.lang.Thread.run(Thread.java:744)
> >>>
> ...
> >>> df.write.format("com.databricks.spark.csv").save("/tmp/d.csv")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/readwriter.py",
line 295, in save
>     self._jwrite.save(path)
>   File "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
line 538, in __call__
>   File "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py",
line 300, in get_return_value
> py4j.protocol.Py4JJavaError: An error occurred while calling o90.save.
> : java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.csv
> 	at scala.sys.package$.error(package.scala:27)
> 	at org.apache.spark.sql.sources.ResolvedDataSource$.lookupDataSource(ddl.scala:216)
> 	at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:302)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:144)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:135)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
> 	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
> 	at py4j.Gateway.invoke(Gateway.java:259)
> 	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
> 	at py4j.commands.CallCommand.execute(CallCommand.java:79)
> 	at py4j.GatewayConnection.run(GatewayConnection.java:207)
> 	at java.lang.Thread.run(Thread.java:744)
> >>>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message