spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Herman van Hovell (JIRA)" <>
Subject [jira] [Resolved] (SPARK-20407) ParquetQuerySuite 'Enabling/disabling ignoreCorruptFiles' flaky test
Date Thu, 20 Apr 2017 16:52:05 GMT


Herman van Hovell resolved SPARK-20407.
       Resolution: Fixed
         Assignee: Bogdan Raducanu
    Fix Version/s: 2.2.0

> ParquetQuerySuite 'Enabling/disabling ignoreCorruptFiles' flaky test
> --------------------------------------------------------------------
>                 Key: SPARK-20407
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 2.2.0
>            Reporter: Bogdan Raducanu
>            Assignee: Bogdan Raducanu
>             Fix For: 2.2.0
> ParquetQuerySuite test "Enabling/disabling ignoreCorruptFiles" can sometimes fail. This
is caused by the fact that when one task fails, the driver call returns and test code continues,
but there might still be tasks running that will be killed at the next killing point.
> There are 2 specific issues created by this:
> 1. Files can be closed some time after the test finishes, so DebugFilesystem.assertNoOpenStreams
fails. One solution for this is to change SharedSqlContext and call assertNoOpenStreams inside
eventually {}
> 2. ParquetFileReader constructor from apache parquet 1.8.2 can leak a stream at line
538. This happens when the next line throws an exception. So, the constructor fails and Spark
doesn't have any way to close the file.
> This happens in this test because the test deletes the temporary directory at the end
(but while tasks might still be running). Deleting the directory causes the constructor to
> The solution for this could be to Thread.sleep at the end of the test or to somehow wait
for all tasks to be definitely killed before finishing the test

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message