Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 24 Jun 2016 20:51:16 +0000 (UTC)
From: "Apache Spark (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12982744.1466801368000.23742.1466801476263@Atlassian.JIRA>
In-Reply-To: <JIRA.12982744.1466801368000@Atlassian.JIRA>
References: <JIRA.12982744.1466801368000@Atlassian.JIRA> <JIRA.12982744.1466801368915@arcas>
Subject: [jira] [Commented] (SPARK-16193) Address flaky
 ExternalAppendOnlyMapSuite spilling tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 24 Jun 2016 20:51:18 -0000


    [ https://issues.apache.org/jira/browse/SPARK-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15348598#comment-15348598 ] 

Apache Spark commented on SPARK-16193:
--------------------------------------

User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/13896

> Address flaky ExternalAppendOnlyMapSuite spilling tests
> -------------------------------------------------------
>
>                 Key: SPARK-16193
>                 URL: https://issues.apache.org/jira/browse/SPARK-16193
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, Tests
>    Affects Versions: 2.0.0
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>            Priority: Minor
>
> We've seen tests fail, for different codecs and operations, like so, most recently from 2.0.0 RC1:
> {code}
> - spilling with compression *** FAILED ***
>   java.lang.Exception: Test failed with compression using codec org.apache.spark.io.LZ4CompressionCodec:
> assertion failed: expected groupByKey to spill, but did not
>   at scala.Predef$.assert(Predef.scala:170)
>   at org.apache.spark.TestUtils$.assertSpilled(TestUtils.scala:170)
>   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite.org$apache$spark$util$collection$ExternalAppendOnlyMapSuite$$testSimpleSpilling(ExternalAppendOnlyMapSuite.scala:253)
>   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:218)
>   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:216)
>   at scala.collection.immutable.Stream.foreach(Stream.scala:594)
>   ...
> {code}
> My theory is that the listener doesn't receive notification of the spilled stages early enough to show up in the test. It should wait until the job is done before letting the test proceed to query the number of spilled stages.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org