spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-17960) Upgrade to Py4J 0.10.4
Date Mon, 17 Oct 2016 14:27:58 GMT


Apache Spark commented on SPARK-17960:

User 'jagadeesanas2' has created a pull request for this issue:

> Upgrade to Py4J 0.10.4
> ----------------------
>                 Key: SPARK-17960
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>            Reporter: holdenk
>            Priority: Trivial
>              Labels: starter
> In general we should try and keep up to date with Py4J's new releases. The changes in
this one are small ( ) and shouldn't
impact Spark in any significant way so I'm going to tag this as a starter issue for someone
looking to get a deeper understanding of how PySpark works.
> Upgrading Py4J can be a bit tricky compared to updating other packages in general the
steps are:
> 1) Upgrade the Py4J version on the Java side
> 2) Update the py4j src zip file we bundle with Spark
> 3) Make sure everything still works (especially the streaming tests because we do weird
things to make streaming work and its the most likely place to break during a Py4J upgrade).
> You can see how these bits have been done in past releases by looking in the git log
for the last time we changed the Py4J version numbers. Sometimes even for "compatible" releases
like this one we may need to make some small code changes in side of PySpark because we hook
into Py4Js internals, but I don't think this should be the case here.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message