spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From buckhx <>
Subject [GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Date Thu, 14 Apr 2016 14:53:49 GMT
GitHub user buckhx opened a pull request:

    [SPARK-5929][PYSPARK] Context addPyPackage and addRequirementsFile

    ## What changes were proposed in this pull request?
    Both of these methods take a package on the master and ship it to the workers when called
instead of having to manually install packages on all workers.
    ## How was this patch tested?
    Unit tests are written, but I do not believe they accurately reflect a distributed environment.
The test_add_py_package is not using addPyPackage and still works. The addRequirementsFile
method requires internet access to hit the global pypi server and won't work on the current
Jenkins build system.
    We have had this patch running at Palantir for about a year in production.

You can merge this pull request into a Git repository by running:

    $ git pull master

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #12398
commit 0ed060df2ec5a1a0427df6c160bd51c7014b29da
Author: buck heroux <>
Date:   2015-02-18T17:50:27Z

    added requirements file to pyspark

commit 6b8bcde60378b58998f5c14d81d72de81f44d718
Author: buck heroux <>
Date:   2015-02-18T23:30:28Z

    tarfile has no contextmanager in python2.

commit 2773483ea6cc244cb7de02c7dc184391a94d29e6
Author: buck heroux <>
Date:   2015-02-19T01:52:06Z

    reqs fix

commit 0371ad9b13f96dcc534d897789ccd32f907d5ed9
Author: buck heroux <>
Date:   2015-02-19T02:06:48Z

    temp tar file

commit f2a46e5d6e309a5ba29259cc1f77e594d932b0f5
Author: buck heroux <>
Date:   2015-03-05T00:51:46Z

    bubbled up try finally

commit fca4be61c6542b807a0d5370f761ef031fc7eb86
Author: buck heroux <>
Date:   2015-03-05T00:53:17Z

    forgot to remove

commit d28752297ec5b10f6afce49713f4efe20f8d533f
Author: buck heroux <>
Date:   2015-04-03T20:35:44Z

    added requirementsFile tests and switch to __import__

commit 76ff63733b9d293c43218aa743a74bcce36a20c9
Author: buck heroux <>
Date:   2015-04-03T20:41:30Z

    merged tests

commit 565bf7fc0889390178f8f96d0ed9a917b287e164
Author: buck heroux <>
Date:   2015-07-14T17:54:12Z

    pep8 styling

commit 23771fde4674c73d5fb0bbb478385d9cf55c648b
Author: buck heroux <>
Date:   2015-10-05T22:07:48Z

    support namespace packages and extract addModule logic

commit cd21c5c06d6a611f4e738199f20582aeb6087fd7
Author: buck heroux <>
Date:   2015-10-05T22:10:17Z

    tmp_dir to mod

commit 39f26d93b7030f2d7bfe7782414dc5b016cdd852
Author: buck heroux <>
Date:   2015-10-21T20:22:06Z

    remove reqs from context constructor

commit 49a4ed0931b20dd2982c16e1ca3533cef65303af
Author: buck heroux <>
Date:   2015-10-21T20:39:24Z

    upstream merge

commit 1501d0f4744d4645a239f3c3fb931469c2038fa8
Author: buckhx <>
Date:   2016-03-31T17:30:58Z

    upstream merge

commit 3af35bb5f54518f2d07db7665c78d18c5fe2bdf2
Author: buck heroux <>
Date:   2016-04-13T19:08:06Z

    Merge remote-tracking branch 'upstream/master'

commit 88a1d6ceffaddfc5f3aa5afc07d65a1d9ddb54e9
Author: buck heroux <>
Date:   2016-04-13T22:57:23Z

    add_py_package test

commit 93b9e9fdd464744f90eb3a7eb325f853e2f113d4
Author: buck heroux <>
Date:   2016-04-14T14:39:17Z

    uncommented pip_requirements test

commit 82476a632c5da724c3ca45ed8b888a946d934104
Author: buck heroux <>
Date:   2016-04-14T14:42:05Z

    removed todo

commit ce9966e875a6d00bbe6325dc5fc9519ce5292c04
Author: buck heroux <>
Date:   2016-04-14T14:47:36Z



If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message