spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From amin mohebbi <aminn_...@yahoo.com.INVALID>
Subject Pyspark Error
Date Tue, 18 Nov 2014 14:10:29 GMT
Hi there,
I have already downloaded Pre-built spark-1.1.0, I want to run pyspark by try typing ./bin/pyspark
but I got the following error:







scala shell is up and working fine

hduser@master:~/Downloads/spark-1.1.0$ ./bin/spark-shell
Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=128m; support was removed
in 8.0
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
.....
.....
14/11/18 04:33:13 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@master:34937/user/HeartbeatReceiver
14/11/18 04:33:13 INFO SparkILoop: Created spark context..
Spark context available as sc.

scala> hduser@master:~/Downloads/spark-1.1.0$ 



But python shell does not work:

hduser@master:~/Downloads/spark-1.1.0$ 
hduser@master:~/Downloads/spark-1.1.0$ 
hduser@master:~/Downloads/spark-1.1.0$ ./bin/pyspark
Python 2.7.3 (default, Feb 27 2014, 20:00:17) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=128m; support was removed
in 8.0
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/11/18 04:36:06 INFO SecurityManager: Changing view acls to: hduser,
14/11/18 04:36:06 INFO SecurityManager: Changing modify acls to: hduser,
14/11/18 04:36:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls
disabled; users with view permissions: Set(hduser, ); users with modify permissions: Set(hduser,
)
14/11/18 04:36:06 INFO Slf4jLogger: Slf4jLogger started
14/11/18 04:36:06 INFO Remoting: Starting remoting
14/11/18 04:36:06 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@master:52317]
14/11/18 04:36:06 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@master:52317]
14/11/18 04:36:06 INFO Utils: Successfully started service 'sparkDriver' on port 52317.
14/11/18 04:36:06 INFO SparkEnv: Registering MapOutputTracker
14/11/18 04:36:06 INFO SparkEnv: Registering BlockManagerMaster
14/11/18 04:36:06 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141118043606-c346
14/11/18 04:36:07 INFO Utils: Successfully started service 'Connection manager for block manager'
on port 47507.
14/11/18 04:36:07 INFO ConnectionManager: Bound socket to port 47507 with id = ConnectionManagerId(master,47507)
14/11/18 04:36:07 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
14/11/18 04:36:07 INFO BlockManagerMaster: Trying to register BlockManager
14/11/18 04:36:07 INFO BlockManagerMasterActor: Registering block manager master:47507 with
267.3 MB RAM
14/11/18 04:36:07 INFO BlockManagerMaster: Registered BlockManager
14/11/18 04:36:07 INFO HttpFileServer: HTTP File server directory is /tmp/spark-8b29544a-c74b-4a3e-88e0-13801c8dcc65
14/11/18 04:36:07 INFO HttpServer: Starting HTTP Server
14/11/18 04:36:07 INFO Utils: Successfully started service 'HTTP file server' on port 40029.
14/11/18 04:36:12 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/11/18 04:36:12 INFO SparkUI: Started SparkUI at http://master:4040
14/11/18 04:36:12 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@master:52317/user/HeartbeatReceiver
14/11/18 04:36:12 INFO SparkUI: Stopped Spark web UI at http://master:4040
14/11/18 04:36:12 INFO DAGScheduler: Stopping DAGScheduler
14/11/18 04:36:13 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
14/11/18 04:36:13 INFO ConnectionManager: Selector thread was interrupted!
14/11/18 04:36:13 INFO ConnectionManager: ConnectionManager stopped
14/11/18 04:36:13 INFO MemoryStore: MemoryStore cleared
14/11/18 04:36:13 INFO BlockManager: BlockManager stopped
14/11/18 04:36:13 INFO BlockManagerMaster: BlockManagerMaster stopped
14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
14/11/18 04:36:13 INFO SparkContext: Successfully stopped SparkContext
14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down;
proceeding with flushing remote transports.
14/11/18 04:36:13 INFO Remoting: Remoting shut down
14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
Traceback (most recent call last):
  File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/shell.py", line 44, in <module>
    sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
  File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/context.py", line 107, in __init__
    conf)
  File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/context.py", line 159, in _do_init
    self._accumulatorServer = accumulators._start_update_server()
  File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/accumulators.py", line 251, in
_start_update_server
    server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
  File "/usr/lib/python2.7/SocketServer.py", line 408, in __init__
    self.server_bind()
  File "/usr/lib/python2.7/SocketServer.py", line 419, in server_bind
    self.socket.bind(self.server_address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.gaierror: [Errno -5] No address associated with hostname
>>> sc.parallelize(range(1000)).count()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sc' is not defined
>>> sc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sc' is not defined
>>> spark
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'spark' is not defined
>>> 
Best Regards

.......................................................

Amin Mohebbi

PhD candidate in Software Engineering 
 at university of Malaysia  

Tel : +60 18 2040 017



E-Mail : TP025921@ex.apiit.edu.my

              amin_524@me.com
Mime
View raw message