spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AlexG <swift...@gmail.com>
Subject importerror using external library with pyspark
Date Thu, 04 Jun 2015 01:24:27 GMT
I have libskylark installed on both machines in my two node cluster in the
same locations, and checked that the following code, which calls libskylark,
works on both nodes with 'pyspark rfmtest.py':

import re
import numpy
import skylark.ml.kernels
import random
import os

from pyspark import SparkContext
sc = SparkContext(appName="test")

SIGMA = 10
NUM_RF = 500
numfeatures = 100
numpoints = 1000
kernel = skylark.ml.kernels.Gaussian(numfeatures, SIGMA)
S = kernel.rft(NUM_RF)

rows = sc.parallelize(numpy.random.rand(numpoints, numfeatures).tolist(), 6)
sketched_rows = rows.map(lambda row : S /
numpy.ndarray(shape=(1,numfeatures), buffer=numpy.array(row)).copy())

os.system("rm -rf spark_out")
sketched_rows.saveAsTextFile('spark_out')

However, when I try to run the same code on the cluster with 'spark-submit
--master spark://master:7077 rfmtest.py', I get an ImportError saying that
skylark.sketch does not exist:

15/06/04 01:21:50 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on master:40244 (size: 67.5 KB, free: 265.3 MB)
15/06/04 01:21:51 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on node001:45690 (size: 67.5 KB, free: 265.3 MB)
15/06/04 01:21:51 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
master): org.apache.spark.api.python.PythonException: Traceback (most recent
call last):
  File "/opt/Spark/python/pyspark/worker.py", line 88, in main
    command = pickleSer._read_with_length(infile)
  File "/opt/Spark/python/pyspark/serializers.py", line 156, in
_read_with_length
    return self.loads(obj)
  File "/opt/Spark/python/pyspark/serializers.py", line 405, in loads
    return cPickle.loads(obj)
ImportError: No module named skylark.sketch

	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:135)
	at
org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:176)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:94)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:64)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Any ideas what might be going on?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/importerror-using-external-library-with-pyspark-tp23145.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message