Mailing-List: contact dev-help@systemml.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.apache.org
From: dusenberrymw@gmail.com
Content-Type: multipart/alternative;
	boundary=Apple-Mail-A2884315-486A-4A7D-AF32-3BB307CA82D6
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)
Date: Wed, 5 Jul 2017 14:23:23 -0700
Subject: Re: Install - Configure Jupyter Notebook
Message-Id: <33AA063C-8D48-4A8B-82F1-C9187E3A2E86@gmail.com>
References: <81F75F6B-7F9D-4432-BDDE-FD90A9CD6018@thinkwrap.com> <MAXPR01MB0220533B51D8CE106E10F835A3D10@MAXPR01MB0220.INDPRD01.PROD.OUTLOOK.COM> <OFAB64F902.4DA72554-ON00258154.0074242D-88258154.0074A80F@notes.na.collabserv.com>
In-Reply-To: <OFAB64F902.4DA72554-ON00258154.0074242D-88258154.0074A80F@notes.na.collabserv.com>
To: dev@systemml.apache.org
archived-at: Wed, 05 Jul 2017 21:23:36 -0000

--Apple-Mail-A2884315-486A-4A7D-AF32-3BB307CA82D6
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

For a bit more context, this is the general way of starting Jupyter with PyS=
park support.  In contrast, the usual `jupyter notebook` command will only l=
aunch Jupyter with a standard Python kernel.

Additionally, all of the extra "conf" settings in that command refer to sett=
ings that could be placed in the standard `conf/spark-defaults.conf`file of y=
our Spark installation, with spaces instead of the equals signs, in case you=
're already familiar with that.

- Mike

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Jul 5, 2017, at 2:14 PM, Niketan Pansare <npansar@us.ibm.com> wrote:
>=20
> Hi Gustavo,
>=20
> You can paste that code into the commandline:
> $ PYSPARK_DRIVER_PYTHON=3Djupyter PYSPARK_DRIVER_PYTHON_OPTS=3D"notebook" p=
yspark --master local[*] --conf "spark.driver.memory=3D12g" --conf spark.dri=
ver.maxResultSize=3D0 --conf spark.akka.frameSize=3D128 --conf spark.default=
.parallelism=3D100
>=20
> The above command tells "pyspark" that the python driver is jupyter. For m=
ore details, please see https://github.com/apache/spark/blob/master/bin/pysp=
ark#L27
>=20
> Alternatively, you can follow Arijit's suggestion.
>=20
> Thanks,
>=20
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=3Dus-npansar
>=20
> arijit chakraborty ---07/02/2017 04:22:28 AM---Hi Gustavo, You can put tha=
t pyspark details in the jupyter console itself.
>=20
> From: arijit chakraborty <akc14@hotmail.com>
> To: "dev@systemml.apache.org" <dev@systemml.apache.org>
> Date: 07/02/2017 04:22 AM
> Subject: Re: Install - Configure Jupyter Notebook
>=20
>=20
>=20
>=20
> Hi Gustavo,
>=20
>=20
> You can put that pyspark details in the jupyter console itself.
>=20
>=20
> import os
> import sys
> import pandas as pd
> import numpy as np
>=20
> spark_path =3D "C:\spark"
> os.environ['SPARK_HOME'] =3D spark_path
> os.environ['HADOOP_HOME'] =3D spark_path
>=20
> sys.path.append(spark_path + "/bin")
> sys.path.append(spark_path + "/python")
> sys.path.append(spark_path + "/python/pyspark/")
> sys.path.append(spark_path + "/python/lib")
> sys.path.append(spark_path + "/python/lib/pyspark.zip")
> sys.path.append(spark_path + "/python/lib/py4j-0.10.4-src.zip")
>=20
> from pyspark import SparkContext
> from pyspark import SparkConf
>=20
> sc =3D SparkContext("local[*]", "test")
>=20
>=20
> # SystemML Specifications:
>=20
>=20
> from pyspark.sql import SQLContext
> import systemml as sml
> sqlCtx =3D SQLContext(sc)
> ml =3D sml.MLContext(sc)
>=20
>=20
> But this is not a very good way of doing it. I did it as I'm using windows=
 and it's easier to do it like that.
>=20
>=20
> Regards,
>=20
> Arijit
>=20
> ________________________________
> From: Gustavo Frederico <gustavo.frederico@thinkwrap.com>
> Sent: Sunday, July 2, 2017 10:16:03 AM
> To: dev@systemml.apache.org
> Subject: Install - Configure Jupyter Notebook
>=20
>=20
> A basic question: step 3 in https://systemml.apache.org/install-systemml.h=
tml <https://systemml.apache.org/install-systemml.html>  for =E2=80=9CConfig=
ure Jupyter Notebook=E2=80=9D has
> # Start Jupyter Notebook Server
> PYSPARK_DRIVER_PYTHON=3Djupyter PYSPARK_DRIVER_PYTHON_OPTS=3D"notebook" py=
spark --master local[*] --conf "spark.driver.memory=3D12g" --conf spark.driv=
er.maxResultSize=3D0 --conf spark.akka.frameSize=3D128 --conf spark.default.=
parallelism=3D100
> Where does that go? There are no details in this step=E2=80=A6
>=20
> Thanks
>=20
> Gustavo
>=20
>=20
>=20

--Apple-Mail-A2884315-486A-4A7D-AF32-3BB307CA82D6--