spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ericl <>
Subject [GitHub] spark pull request #15306: [SPARK-17740] Spark tests should mock / interpose...
Date Fri, 30 Sep 2016 01:50:12 GMT
GitHub user ericl opened a pull request:

    [SPARK-17740] Spark tests should mock / interpose HDFS to ensure that streams are closed

    ## What changes were proposed in this pull request?
    As a followup to SPARK-17666, ensure filesystem connections are not leaked at least in
unit tests. This is done here by intercepting filesystem calls as suggested by @JoshRosen
. At the end of each test, we assert no filesystem streams are left open.
    This applies to all tests using SharedSQLContext or SharedSparkContext.
    ## How was this patch tested?
    I verified that tests in sql and core are indeed using the filesystem backend, and fixed
the detected leaks. I also checked that reverting
causes many more test failures due to connection leaks.

You can merge this pull request into a Git repository by running:

    $ git pull sc-4672

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15306
commit c07b6136d2d80595612e932325b7d47ae5f36f56
Author: Eric Liang <>
Date:   2016-09-30T01:23:57Z

    Thu Sep 29 18:23:57 PDT 2016

commit a92de9626929cc4907ac789f4dd49336d0e103bd
Author: Eric Liang <>
Date:   2016-09-30T01:33:52Z

    Thu Sep 29 18:33:52 PDT 2016

commit c009783f5846669e10578492114a1a0d6631e611
Author: Eric Liang <>
Date:   2016-09-30T01:42:03Z

    Thu Sep 29 18:42:03 PDT 2016

commit 1d80d2ec396bc5225da351ff0c7e143c79000c83
Author: Eric Liang <>
Date:   2016-09-30T01:46:00Z

    Thu Sep 29 18:46:00 PDT 2016


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message