spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <>
Subject [jira] [Commented] (SPARK-12243) PySpark tests are slow in Jenkins
Date Mon, 07 Mar 2016 17:00:41 GMT


Dongjoon Hyun commented on SPARK-12243:

Tests passed in 763 seconds
We saves about 200 seconds.

> PySpark tests are slow in Jenkins
> ---------------------------------
>                 Key: SPARK-12243
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Project Infra, PySpark, Tests
>            Reporter: Josh Rosen
> In the Jenkins pull request builder, it looks like PySpark tests take around 992 seconds
(~16.5 minutes) of end-to-end time to run, despite the fact that we run four Python test suites
in parallel. We should try to figure out why this is slow and see if there's any easy way
to speed things up.
> Note that the PySpark streaming tests take about 5 minutes to run, so best-case we're
looking at a 10 minute speedup via further parallelization. We should also try to see whether
there are individual slow tests in those Python suites which can be sped up or skipped.
> We could also consider running only the Python 2.6 tests in non-Pyspark pull request
builds and reserve testing of all Python versions for builds which touch PySpark-related code.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message