spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <>
Subject [jira] [Created] (SPARK-12243) PySpark tests are slow in Jenkins
Date Wed, 09 Dec 2015 19:39:11 GMT
Josh Rosen created SPARK-12243:

             Summary: PySpark tests are slow in Jenkins
                 Key: SPARK-12243
             Project: Spark
          Issue Type: Sub-task
          Components: Project Infra, PySpark, Tests
            Reporter: Josh Rosen

In the Jenkins pull request builder, it looks like PySpark tests take around 992 seconds (~16.5
minutes) of end-to-end time to run, despite the fact that we run four Python test suites in
parallel. We should try to figure out why this is slow and see if there's any easy way to
speed things up.

Note that the PySpark streaming tests take about 5 minutes to run, so best-case we're looking
at a 10 minute speedup via further parallelization. We should also try to see whether there
are individual slow tests in those Python suites which can be sped up or skipped.

We could also consider running only the Python 2.6 tests in non-Pyspark pull request builds
and reserve testing of all Python versions for builds which touch PySpark-related code.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message