spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <>
Subject [jira] [Commented] (SPARK-3534) Avoid running MLlib and Streaming tests when testing SQL PRs
Date Mon, 15 Sep 2014 18:42:35 GMT


Josh Rosen commented on SPARK-3534:

Looks like this has been proposed before: SPARK-1455

> Avoid running MLlib and Streaming tests when testing SQL PRs
> ------------------------------------------------------------
>                 Key: SPARK-3534
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Project Infra, SQL
>            Reporter: Michael Armbrust
>            Priority: Blocker
> We are bumping up against the 120 minute time limit for tests pretty regularly now. 
Since we have decreased the number of shuffle partitions and up-ed the parallelism I don't
think there is much low hanging fruit to speed up the SQL tests. (The tests that are listed
as taking 2-3 minutes are actually 100s of tests that I think are valuable).  Instead I propose
we avoid running tests that we don't need to.
> This will have the added benefit of eliminating failures in SQL due to flaky streaming
> Note that this won't fix the full builds that are run for every commit.  There I think
we just just up the test timeout.
> cc: [~joshrosen] [~pwendell]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message