Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (athena.apache.org: domain of binwang.cu@gmail.com
 designates 74.125.82.43 as permitted sender)
MIME-Version: 1.0
Date: Sun, 28 Dec 2014 11:57:31 -0700
Message-ID: 
 <CAJHCcVbPe05X6+6moHywR8nrmk54V9eveg4edG5Fa+x79bdujQ@mail.gmail.com>
Subject: Anaconda iPython notebook working with CDH Spark
From: Bin Wang <binwang.cu@gmail.com>
To: "Apache.Spark.User" <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=047d7b86deae8307b0050b4b54c0

--047d7b86deae8307b0050b4b54c0
Content-Type: text/plain; charset=UTF-8

Hi there,

I have a cluster with CDH5.1 running on top of Redhat6.5, where the default
Python version is 2.6. I am trying to set up a proper iPython notebook
environment to develop spark application using pyspark.

Here
<http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/>
is a tutorial that I have been following. However, it turned out that the
author was using iPython1 where we have the latest Anaconda Python2.7
installed on our name node. When I finished following the tutorial, I can
connect to the spark cluster but whenever I tried to distribute the work,
it will errorred out and google tells me it is the difference between the
version of Python across the cluster.

Here are a few thoughts that I am planning to try.
(1) remove the Anaconda Python from the namenode and install the iPython
version that is compatible with Python2.6.
(2) or I need to install Anaconda Python on every node and make it the
default Python version across the whole cluster (however, I am not sure if
this plan will totally screw up the existing environment since some running
services are built by Python2.6...)

Let me which should be the proper way to set up an iPython notebook
environment.

Best regards,

Bin

--047d7b86deae8307b0050b4b54c0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi there,=C2=A0<div><br></div><div>I have a cluster with C=
DH5.1 running on top of Redhat6.5, where the default Python version is 2.6.=
 I am trying to set up a proper iPython notebook environment to develop spa=
rk application using pyspark.=C2=A0</div><div><br></div><div><a href=3D"htt=
p://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-=
spark/">Here</a> is a tutorial that I have been following. However, it turn=
ed out that the author was using iPython1 where we have the latest Anaconda=
 Python2.7 installed on our name node. When I finished following the tutori=
al, I can connect to the spark cluster but whenever I tried to distribute t=
he work, it will errorred out and google tells me it is the difference betw=
een the version of Python across the cluster.=C2=A0</div><div><br></div><di=
v>Here are a few thoughts that I am planning to try.=C2=A0</div><div>(1) re=
move the Anaconda Python from the namenode and install the iPython version =
that is compatible with Python2.6.=C2=A0</div><div>(2) or I need to install=
 Anaconda Python on every node and make it the default Python version acros=
s the whole cluster (however, I am not sure if this plan will totally screw=
 up the existing environment since some running services are built by Pytho=
n2.6...)</div><div><br></div><div>Let me which should be the proper way to =
set up an iPython notebook environment.</div><div><br></div><div>Best regar=
ds,=C2=A0</div><div><br></div><div>Bin</div></div>

--047d7b86deae8307b0050b4b54c0--