mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stevo Slavic (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAHOUT-1325) MapReduce unit tests cannot be run in parallel
Date Mon, 28 Oct 2013 13:53:31 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13806770#comment-13806770
] 

Stevo Slavic commented on MAHOUT-1325:
--------------------------------------

[~isabel], I've applied your patch, but M/R tests are still failing.
Here is relevant build output from Mahout-Trunk build job failed run #1852:
{noformat}
[INFO] ------------------------------------------------------------------------
[INFO] Building Mahout Integration 0.9-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ mahout-integration ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mahout-integration ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ mahout-integration ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 127 source files to /x1/jenkins/jenkins-slave/workspace/Mahout-Trunk/trunk/integration/target/classes
[WARNING] Note: /x1/jenkins/jenkins-slave/workspace/Mahout-Trunk/trunk/integration/src/main/java/org/apache/mahout/cf/taste/impl/model/mongodb/MongoDBDataModel.java
uses or overrides a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[WARNING] Note: /x1/jenkins/jenkins-slave/workspace/Mahout-Trunk/trunk/integration/src/main/java/org/apache/mahout/cf/taste/impl/model/mongodb/MongoDBDataModel.java
uses unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ mahout-integration
---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 10 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ mahout-integration
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 37 source files to /x1/jenkins/jenkins-slave/workspace/Mahout-Trunk/trunk/integration/target/test-classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[WARNING] Note: Some input files use unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ mahout-integration ---
[INFO] Surefire report directory: /x1/jenkins/jenkins-slave/workspace/Mahout-Trunk/trunk/integration/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.mahout.utils.regex.RegexMapperTest
Running org.apache.mahout.utils.email.MailProcessorTest
Running org.apache.mahout.text.LuceneSegmentInputFormatTest
Running org.apache.mahout.utils.TestConcatenateVectorsJob
Running org.apache.mahout.clustering.TestClusterDumper
Running org.apache.mahout.utils.vectors.arff.MapBackedARFFModelTest
Running org.apache.mahout.utils.vectors.arff.ARFFVectorIterableTest
Running org.apache.mahout.utils.vectors.csv.CSVVectorIteratorTest
Running org.apache.mahout.utils.SplitInputTest
Running org.apache.mahout.text.LuceneSegmentRecordReaderTest
Running org.apache.mahout.text.SequenceFilesFromLuceneStorageMRJobTest
Running org.apache.mahout.utils.vectors.arff.ARFFTypeTest
Running org.apache.mahout.text.LuceneStorageConfigurationTest
Running org.apache.mahout.text.LuceneSegmentInputSplitTest
Running org.apache.mahout.text.SequenceFilesFromLuceneStorageDriverTest
Running org.apache.mahout.text.MailArchivesClusteringAnalyzerTest
Running org.apache.mahout.clustering.TestClusterEvaluator
Running org.apache.mahout.utils.regex.RegexUtilsTest
Running org.apache.mahout.cf.taste.impl.similarity.jdbc.MySQLJDBCInMemoryItemSimilarityTest
Running org.apache.mahout.utils.vectors.VectorHelperTest
Running org.apache.mahout.utils.vectors.arff.DriverTest
Running org.apache.mahout.clustering.cdbw.TestCDbwEvaluator
Running org.apache.mahout.utils.nlp.collocations.llr.BloomTokenFilterTest
Running org.apache.mahout.utils.vectors.io.VectorWriterTest
Running org.apache.mahout.utils.vectors.lucene.LuceneIterableTest
Running org.apache.mahout.text.SequenceFilesFromLuceneStorageTest
Running org.apache.mahout.utils.vectors.lucene.DriverTest
Running org.apache.mahout.utils.Bump125Test
Running org.apache.mahout.text.SequenceFilesFromMailArchivesTest
Running org.apache.mahout.utils.vectors.lucene.CachedTermInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.2 sec - in org.apache.mahout.utils.vectors.arff.MapBackedARFFModelTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.249 sec - in org.apache.mahout.utils.vectors.arff.ARFFTypeTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.187 sec - in org.apache.mahout.utils.Bump125Test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.278 sec - in org.apache.mahout.utils.regex.RegexUtilsTest
Running org.apache.mahout.text.TestSequenceFilesFromDirectory
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.406 sec - in org.apache.mahout.utils.vectors.arff.ARFFVectorIterableTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.397 sec - in org.apache.mahout.text.MailArchivesClusteringAnalyzerTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.386 sec - in org.apache.mahout.utils.vectors.VectorHelperTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.388 sec - in org.apache.mahout.utils.nlp.collocations.llr.BloomTokenFilterTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.592 sec - in org.apache.mahout.utils.email.MailProcessorTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.673 sec - in org.apache.mahout.utils.vectors.csv.CSVVectorIteratorTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.664 sec - in org.apache.mahout.cf.taste.impl.similarity.jdbc.MySQLJDBCInMemoryItemSimilarityTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.79 sec - in org.apache.mahout.utils.vectors.arff.DriverTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.948 sec - in org.apache.mahout.text.LuceneStorageConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.987 sec - in org.apache.mahout.utils.vectors.lucene.CachedTermInfoTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.357 sec - in org.apache.mahout.utils.regex.RegexMapperTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.282 sec - in org.apache.mahout.utils.TestConcatenateVectorsJob
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.247 sec - in org.apache.mahout.utils.vectors.lucene.LuceneIterableTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.455 sec - in org.apache.mahout.utils.vectors.io.VectorWriterTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.99 sec - in org.apache.mahout.text.LuceneSegmentInputFormatTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.92 sec - in org.apache.mahout.text.LuceneSegmentInputSplitTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.012 sec - in org.apache.mahout.utils.vectors.lucene.DriverTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.17 sec - in org.apache.mahout.text.LuceneSegmentRecordReaderTest
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.735 sec <<< FAILURE!
- in org.apache.mahout.text.SequenceFilesFromMailArchivesTest
testMapReduce(org.apache.mahout.text.SequenceFilesFromMailArchivesTest)  Time elapsed: 1.368
sec  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.junit.Assert.assertEquals(Assert.java:542)
	at org.apache.mahout.text.SequenceFilesFromMailArchivesTest.testMapReduce(SequenceFilesFromMailArchivesTest.java:153)

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.317 sec - in org.apache.mahout.text.SequenceFilesFromLuceneStorageTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.689 sec - in org.apache.mahout.text.SequenceFilesFromLuceneStorageMRJobTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.754 sec - in org.apache.mahout.text.TestSequenceFilesFromDirectory
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.409 sec - in org.apache.mahout.text.SequenceFilesFromLuceneStorageDriverTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.987 sec - in org.apache.mahout.clustering.TestClusterDumper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.471 sec - in org.apache.mahout.clustering.cdbw.TestCDbwEvaluator
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.861 sec - in org.apache.mahout.utils.SplitInputTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.891 sec - in org.apache.mahout.clustering.TestClusterEvaluator

Results :

Failed tests: 
  SequenceFilesFromMailArchivesTest.testMapReduce:153->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
expected:<1> but was:<0>

Tests run: 95, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Mahout Build Tools ................................ SUCCESS [3.161s]
[INFO] Apache Mahout ..................................... SUCCESS [1.253s]
[INFO] Mahout Math ....................................... SUCCESS [24.020s]
[INFO] Mahout Core ....................................... SUCCESS [1:56.245s]
[INFO] Mahout Integration ................................ FAILURE [18.384s]
[INFO] Mahout Examples ................................... SKIPPED
[INFO] Mahout Release Package ............................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:43.563s
[INFO] Finished at: Mon Oct 28 09:39:41 UTC 2013
[INFO] Final Memory: 35M/333M
[INFO] ------------------------------------------------------------------------
{noformat}

Besides this patch, I believe we still need fix for MAPREDUCE-5367 so that Mahout M/R tests
can run in parallel.

> MapReduce unit tests cannot be run in parallel
> ----------------------------------------------
>
>                 Key: MAHOUT-1325
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-1325
>             Project: Mahout
>          Issue Type: Bug
>          Components: Integration
>    Affects Versions: 0.8
>         Environment: ubuntu1 Apache Jenkins CI node
>            Reporter: Stevo Slavic
>            Assignee: Stevo Slavic
>            Priority: Minor
>             Fix For: 0.9
>
>         Attachments: MAHOUT-1325.patch, org.apache.mahout.text.SequenceFilesFromMailArchivesTest-output.txt
>
>
> Mahout-Quality build job execution #2225 failed because of this test.
> Relevant build log:
> {noformat}
> [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ mahout-integration ---
> [INFO] Surefire report directory: /home/jenkins/jenkins-slave/workspace/Mahout-Quality/trunk/integration/target/surefire-reports
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> Running org.apache.mahout.utils.vectors.lucene.LuceneIterableTest
> Running org.apache.mahout.utils.email.MailProcessorTest
> Running org.apache.mahout.clustering.cdbw.TestCDbwEvaluator
> Running org.apache.mahout.utils.regex.RegexMapperTest
> Running org.apache.mahout.utils.vectors.io.VectorWriterTest
> Running org.apache.mahout.utils.nlp.collocations.llr.BloomTokenFilterTest
> Running org.apache.mahout.utils.vectors.lucene.CachedTermInfoTest
> Running org.apache.mahout.clustering.TestClusterEvaluator
> Running org.apache.mahout.utils.regex.RegexUtilsTest
> Running org.apache.mahout.utils.vectors.lucene.DriverTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 sec - in org.apache.mahout.utils.regex.RegexUtilsTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.412 sec - in org.apache.mahout.utils.email.MailProcessorTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.344 sec - in org.apache.mahout.utils.nlp.collocations.llr.BloomTokenFilterTest
> Running org.apache.mahout.utils.vectors.csv.CSVVectorIteratorTest
> Running org.apache.mahout.clustering.TestClusterDumper
> Running org.apache.mahout.utils.vectors.VectorHelperTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.601 sec - in org.apache.mahout.utils.vectors.csv.CSVVectorIteratorTest
> Running org.apache.mahout.utils.vectors.arff.ARFFTypeTest
> Running org.apache.mahout.utils.vectors.arff.ARFFVectorIterableTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.383 sec - in org.apache.mahout.utils.vectors.VectorHelperTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.332 sec - in org.apache.mahout.utils.vectors.lucene.LuceneIterableTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec - in org.apache.mahout.utils.vectors.arff.ARFFTypeTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.369 sec - in org.apache.mahout.utils.vectors.lucene.CachedTermInfoTest
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.415 sec - in org.apache.mahout.utils.vectors.arff.ARFFVectorIterableTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.5 sec - in org.apache.mahout.utils.regex.RegexMapperTest
> Running org.apache.mahout.utils.vectors.arff.DriverTest
> Running org.apache.mahout.utils.Bump125Test
> Running org.apache.mahout.utils.SplitInputTest
> Running org.apache.mahout.utils.TestConcatenateVectorsJob
> Running org.apache.mahout.utils.vectors.arff.MapBackedARFFModelTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.043 sec - in org.apache.mahout.utils.vectors.io.VectorWriterTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.362 sec - in org.apache.mahout.utils.Bump125Test
> Running org.apache.mahout.cf.taste.impl.similarity.jdbc.MySQLJDBCInMemoryItemSimilarityTest
> Running org.apache.mahout.text.SequenceFilesFromLuceneStorageTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.304 sec - in org.apache.mahout.utils.vectors.arff.MapBackedARFFModelTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.887 sec - in org.apache.mahout.utils.vectors.arff.DriverTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.486 sec - in org.apache.mahout.cf.taste.impl.similarity.jdbc.MySQLJDBCInMemoryItemSimilarityTest
> Running org.apache.mahout.text.TestSequenceFilesFromDirectory
> Running org.apache.mahout.text.LuceneStorageConfigurationTest
> Running org.apache.mahout.text.LuceneSegmentInputFormatTest
> Running org.apache.mahout.text.SequenceFilesFromMailArchivesTest
> Running org.apache.mahout.text.MailArchivesClusteringAnalyzerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.265 sec - in org.apache.mahout.text.MailArchivesClusteringAnalyzerTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.8 sec - in org.apache.mahout.utils.TestConcatenateVectorsJob
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.097 sec - in org.apache.mahout.text.LuceneStorageConfigurationTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.902 sec - in org.apache.mahout.utils.vectors.lucene.DriverTest
> Running org.apache.mahout.text.LuceneSegmentRecordReaderTest
> Running org.apache.mahout.text.LuceneSegmentInputSplitTest
> Running org.apache.mahout.text.SequenceFilesFromLuceneStorageMRJobTest
> Running org.apache.mahout.text.SequenceFilesFromLuceneStorageDriverTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.237 sec - in org.apache.mahout.text.LuceneSegmentInputFormatTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.122 sec <<<
FAILURE! - in org.apache.mahout.text.SequenceFilesFromMailArchivesTest
> testMapReduce(org.apache.mahout.text.SequenceFilesFromMailArchivesTest)  Time elapsed:
2.333 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.failNotEquals(Assert.java:743)
> 	at org.junit.Assert.assertEquals(Assert.java:118)
> 	at org.junit.Assert.assertEquals(Assert.java:555)
> 	at org.junit.Assert.assertEquals(Assert.java:542)
> 	at org.apache.mahout.text.SequenceFilesFromMailArchivesTest.testMapReduce(SequenceFilesFromMailArchivesTest.java:153)
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.74 sec - in org.apache.mahout.text.TestSequenceFilesFromDirectory
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.172 sec - in org.apache.mahout.text.LuceneSegmentInputSplitTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.85 sec - in org.apache.mahout.text.LuceneSegmentRecordReaderTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.571 sec - in org.apache.mahout.clustering.TestClusterDumper
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.021 sec - in org.apache.mahout.text.SequenceFilesFromLuceneStorageTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.121 sec - in org.apache.mahout.text.SequenceFilesFromLuceneStorageMRJobTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.613 sec - in org.apache.mahout.text.SequenceFilesFromLuceneStorageDriverTest
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.246 sec - in org.apache.mahout.utils.SplitInputTest
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.07 sec - in org.apache.mahout.clustering.cdbw.TestCDbwEvaluator
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.224 sec - in org.apache.mahout.clustering.TestClusterEvaluator
> Results :
> Failed tests: 
>   SequenceFilesFromMailArchivesTest.testMapReduce:153->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
expected:<1> but was:<0>
> Tests run: 95, Failures: 1, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Mahout Build Tools ................................ SUCCESS [8.296s]
> [INFO] Apache Mahout ..................................... SUCCESS [2.731s]
> [INFO] Mahout Math ....................................... SUCCESS [1:38.594s]
> [INFO] Mahout Core ....................................... SUCCESS [4:44.809s]
> [INFO] Mahout Integration ................................ FAILURE [26.458s]
> [INFO] Mahout Examples ................................... SKIPPED
> [INFO] Mahout Release Package ............................ SKIPPED
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 7:03.463s
> [INFO] Finished at: Tue Sep 03 22:40:41 UTC 2013
> [INFO] Final Memory: 62M/447M
> [INFO] ------------------------------------------------------------------------
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message