spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saikat Kanjilal (JIRA)" <>
Subject [jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests
Date Sat, 10 Dec 2016 20:14:58 GMT


Saikat Kanjilal commented on SPARK-9487:

I'm ok closing it actually but it does outline issues with robustness around the unit tests,
should we open up another jira or reframe this effort to make the unit tests more robust,
that may require some more thought/redesign to produce identical results locally as well as
in jenkins, my vote would be to close this out and recreate another jira that I can take on
to make the unit tests more robust for 1 specific component with very narrowly defined goals,
what do you think?

> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>                 Key: SPARK-9487
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Spark Core, SQL, Tests
>    Affects Versions: 1.5.0
>            Reporter: Xiangrui Meng
>              Labels: starter
>         Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
> In Python we use `local[4]` for unit tests, while in Scala/Java we use `local[2]` and
`local` for some unit tests in SQL, MLLib, and other components. If the operation depends
on partition IDs, e.g., random number generator, this will lead to different result in Python
and Scala/Java. It would be nice to use the same number in all unit tests.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message