I think this list is no longer in use. Instead just use the main maven-dev list. -Dan
I am the author of the patches in http://jira.codehaus.org/browse/SUREFIRE-555. I was wondering if anyone would have the time to review this patch and hopefully accept it for inclusion in surefire. I am more than willing to make further adjustments if necessary. The patch initially had some external dependencies that were a bit complicated, and these have been removed. The only known issue with this patch at the moment is that the execution times for the tests are totally incorrect. This will require a bigger rewrite of the reporting module. (I can do this as a separate patch if needed, but this patch is large enough already) Regards, Kristian Rosenvold
Hi, I have a test integration profile in the project's main pom, where I am trying to override the testSourceDirectory specified in the project tag. The surefire ignores the new value of the testSourceDirectory. Please, help. Vicki -- View this message in context: http://www.nabble.com/Surefilre-configuration-in-profile-won%27t-override-testSourceDirectory-tp18492320p18492320.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
Hi, I am using surefire to run Junit's 4.x parametrerized tests. Surefire reports them as 1 test. If one test out of the parameterized bundle fails - Surefire fails the entire encapsulating test. Is there a way to set the Surefire in such a way that it reports all the parameterized tests? Thanks, Vicki -- View this message in context: http://www.nabble.com/Surefire-reports-on-Junit-Parameterized-tests-tp18412308p18412308.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
roadtripryan wrote: > I am trying to test using TestNG/Spring/ and Maven Surefire 2.4.1. > > My test suite runs great from within eclipse as individual tests or as a > whole suite. When I try and run the tests in Maven Surefire, however, they > fail. It appears Surefire is not calling the correct @BeforeMethod method. > (When running the whole suite, or an individual test.) As you might expect, all Surefire does is call TestNG directly; we don't directly call @BeforeMethod, @Test, etc. > From testing it, it seems Surefire calls the @BeforeMethod in the base > class, but not in extending classes if they override it. I am extending > Springs AbstractTestNGSpringContextTests, and it is calling the base class > methods, not my extending classes @BeforeMethod and @AfterMethod. > Interestingly, it seems to call the correct @BeforeClass method. If you're having a problem with a TestNG test that you believe should be working, please file a bug on jira.codehaus.org; please also attach a minimal Maven project that reproduces the problem. Ideally a test like this should be nothing more than a pom.xml file and a couple of TestNG classes in appropriate directories. (Please don't include any spring classes in this test case unless it's absolutely necessary to do so.) -Dan
I am trying to test using TestNG/Spring/ and Maven Surefire 2.4.1. My test suite runs great from within eclipse as individual tests or as a whole suite. When I try and run the tests in Maven Surefire, however, they fail. It appears Surefire is not calling the correct @BeforeMethod method. (When running the whole suite, or an individual test.) >From testing it, it seems Surefire calls the @BeforeMethod in the base class, but not in extending classes if they override it. I am extending Springs AbstractTestNGSpringContextTests, and it is calling the base class methods, not my extending classes @BeforeMethod and @AfterMethod. Interestingly, it seems to call the correct @BeforeClass method. Any ideas? -- View this message in context: http://www.nabble.com/TestNG-%40BeforeMethod-not-working-on-subclasses.-tp15418799p15418799.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
brettporter wrote: > > I don't know why people even use it with JUnit > I am not really after forking all my tests, they already take long enough... As Dan already guessed, I was more after a way to hack me through to single-class test-suites. If forkMode=always would have worked, I would have argued, "well, I want this behavior but without the JVM forking, please" ;-) Benjamin Bentmann -- View this message in context: http://www.nabble.com/testng%2C-surefire%2C-and-forkMode%3Dalways-tp15323782p15324832.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
dfabulich wrote: > > What should I do here? Put in the hack and let the user figure out the > TestNGException if they try to use multi-class dependent methods? Please, give us the exception from TestNG if the setup is invalid, give us the mere test results if the setup is valid because no inter-class dependencies exist. I mean somebody who can setup a config file for a build tool is smart enough to figure out what an exception means or do you disagree? Just in case, a few lines of doc about the forkMode=always setting should successfully shield Surefire from users filling in unjustified issues. To me, it seems this discussion is primarily concerned about making Surefire bullet-proof against bad user config. As for me, I have this slightly arrogant attitude to feel superior to a machine and hence always feel kind of angry if a program wants to tell what I should better not do because it might fail. Benjamin Bentmann -- View this message in context: http://www.nabble.com/testng%2C-surefire%2C-and-forkMode%3Dalways-tp15323782p15324776.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
Cédric Beust ♔ wrote: > > Creating a TestNG instance for every @Test method? Seems completely > useless > to me. > The intent is not to fork once per test method but once per test class. Just like I can say in the TestNG Eclipse plugin "run class XY" instead of selecting a group. Benjamin Bentmann -- View this message in context: http://www.nabble.com/testng%2C-surefire%2C-and-forkMode%3Dalways-tp15323782p15324537.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
My inclination would be towards just an error that says its not supported under TestNG. I don't know why people even use it with JUnit, except when they have memory leaks in their tests, or tests that affect the state of the JVM that they don't tear down :) - Brett On 07/02/2008, at 9:59 AM, Dan Fabulich wrote: > > Benjamin Bentmann has filed bug SUREFIRE-446 http://jira.codehaus.org/browse/SUREFIRE-446 > > Maven Surefire provides a facility for various "fork modes": > > * forkMode=once: The default mode, spawns a separate TestNG process > (like the TestNG Ant task) > > * forkMode=never: run TestNG in the same process as Maven, in an > isolated classloader > > * forkMode=always: launch TestNG N times for N tests; if one test > segfaults or calls System.exit or something, the other tests can run > to completion safely. > > I didn't test forkMode=always with TestNG in Surefire 2.4, and sure > enough it doesn't work at all. (It runs the tests but reports 0 > tests passed/failed.) > > It does work basically OK for JUnit, where classes are normally > isolated from each other, but I'm not sure it even makes sense for > TestNG tests, where one test may depend on tests in other classes > (multi-class dependent methods). > > I can kind of hack it so it works OK if you don't have multi-class > dependent methods, but if you DO have multi-class dependent methods, > it fails with a TestNGException: "MyTest.testFoo() is depending on > nonexistent method otherClassSetUp". > > What should I do here? Put in the hack and let the user figure out > the TestNGException if they try to use multi-class dependent > methods? Fail with a clean error if the user tries to use > forkMode=always with TestNG? Something else? > > Should forkMode=always be a feature of TestNG somehow? > > -Dan
Cédric Beust ♔ wrote: > Creating a TestNG instance for every @Test method? Seems completely useless > to me. No, once for every test class. > Especially when you can control the threading behavior with the parallel > attribute. Well, the point is crash protection, not threading. (Also, Benjamin would like to keep separate XML files for every class, separate stdout files for every class, etc.) -Dan
Creating a TestNG instance for every @Test method? Seems completely useless to me. Especially when you can control the threading behavior with the parallel attribute. -- Cédric On Wed, Feb 6, 2008 at 2:59 PM, Dan Fabulich <dan@fabulich.com> wrote: > > > Benjamin Bentmann has filed bug SUREFIRE-446 > http://jira.codehaus.org/browse/SUREFIRE-446 > > Maven Surefire provides a facility for various "fork modes": > > * forkMode=once: The default mode, spawns a separate TestNG process (like > the TestNG Ant task) > > * forkMode=never: run TestNG in the same process as Maven, in an isolated > classloader > > * forkMode=always: launch TestNG N times for N tests; if one test > segfaults or calls System.exit or something, the other tests can run to > completion safely. > > I didn't test forkMode=always with TestNG in Surefire 2.4, and sure enough > it doesn't work at all. (It runs the tests but reports 0 tests > passed/failed.) > > It does work basically OK for JUnit, where classes are normally isolated > from each other, but I'm not sure it even makes sense for TestNG tests, > where one test may depend on tests in other classes (multi-class dependent > methods). > > I can kind of hack it so it works OK if you don't have multi-class > dependent methods, but if you DO have multi-class dependent methods, it > fails with a TestNGException: "MyTest.testFoo() is depending on > nonexistent method otherClassSetUp". > > What should I do here? Put in the hack and let the user figure out the > TestNGException if they try to use multi-class dependent methods? Fail > with a clean error if the user tries to use forkMode=always with TestNG? > Something else? > > Should forkMode=always be a feature of TestNG somehow? > > -Dan > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google Groups > "testng-users" group. > To post to this group, send email to testng-users@googlegroups.com > To unsubscribe from this group, send email to > testng-users-unsubscribe@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/testng-users?hl=en > -~----------~----~----~----~------~----~------~--~--- > >
Benjamin Bentmann has filed bug SUREFIRE-446 http://jira.codehaus.org/browse/SUREFIRE-446 Maven Surefire provides a facility for various "fork modes": * forkMode=once: The default mode, spawns a separate TestNG process (like the TestNG Ant task) * forkMode=never: run TestNG in the same process as Maven, in an isolated classloader * forkMode=always: launch TestNG N times for N tests; if one test segfaults or calls System.exit or something, the other tests can run to completion safely. I didn't test forkMode=always with TestNG in Surefire 2.4, and sure enough it doesn't work at all. (It runs the tests but reports 0 tests passed/failed.) It does work basically OK for JUnit, where classes are normally isolated from each other, but I'm not sure it even makes sense for TestNG tests, where one test may depend on tests in other classes (multi-class dependent methods). I can kind of hack it so it works OK if you don't have multi-class dependent methods, but if you DO have multi-class dependent methods, it fails with a TestNGException: "MyTest.testFoo() is depending on nonexistent method otherClassSetUp". What should I do here? Put in the hack and let the user figure out the TestNGException if they try to use multi-class dependent methods? Fail with a clean error if the user tries to use forkMode=always with TestNG? Something else? Should forkMode=always be a feature of TestNG somehow? -Dan
Benjamin Bentmann wrote: > TestNG is not designed only for the fancy tests, it can run the simple unit > tests, too. Dan, to me it looks you/Surefire are pushing something like "If > you want unit tests, use JUnit, if you want function/integration tests, use > TestNG" by punishing those that employ TestNG for the simple use-case. I am > neither a JUnit nor TestNG fanatic but I can't follow your XOR-like argument > here. You CAN use TestNG for unit tests; the simple use case does work. It's just not as nice as you might wish it were. My argument is that this isn't very important because there's a better alternative available for unit tests. -Dan
dfabulich wrote: > > As I said, I don't think you're using the right tool for the job here. > Hm, I am not really in the mood to start off a discussion with my project manager ;-) dfabulich wrote: > > You can get the exact behavior you want from JUnit. > And I could get it from TestNG if I passed each class in as an individual suite, couldn't I? dfabulich wrote: > > Again, if you want class-level logging, I encourage you to take up the > matter with the TestNG team. > Might be a good idea. As some kind of explanation why I started this discussion on the Surefire corner at all: Surefire 2.3.1 did what I wanted so the first impression after updating to Surefire 2.4 was that it got messed up. dfabulich wrote: > > Doesn't this option smell bad to you? > It's a rather technical/sophisticated option, I agree about that. But aren't "useSystemClassLoader", "childDelegation" and "forkMode" likewise? Whoever develops tests needs to have proper knowledge about their requirements on the execution runtime and hence, needs to configure the test runner appropriately. The same would apply for this option, default it to "all", let the experts change it to meet their special needs. dfabulich wrote: > > Instead of making this option, wouldn't it be nice if it just logged > correctly? If TestNG notified us when to log, we could log anything we > wanted. > I fear having just another lister in the TestNG framework is not enough. The problem I see here is the following: If Sufefire passes all test classes (say A and B) as a single suite to TestNG, what would TestNG prevent from running them like this: A.testOne B.testOne A.testTwo B.testTwo A.testThree B.testThree Our test methods have no dependencies that would require a deterministic order. Even if TestNG exposed events for starting/stopping a test class, I still would not get a guarantee that all methods from a single class are run without inter-leaving with methods from other classes to give me that class-level testing feeling. I hardly can see an argument how one could request a change on this behavior. A suite is the outer-most group of tests and as long as I do not specify dependencies, TestNG is free in ordering them. dfabulich wrote: > > Surefire is meant to be a wrapper around the underlying test framework; > No problems with that. But it's Surefire and not the test framework that aggregates my test classes into a single suite. I am basically asking for an option to control the way how Surefire assembles the suites *before* it passes control to the test framework. Benjamin Bentmann -- View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15138325.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
dfabulich wrote: > > Inter-class dependencies are one of TestNG's fundamental features > Use of this feature is optional. For my sake, call it an "abuse" of TestNG if I run that simple unit tests without inter-class dependencies ;-) dfabulich wrote: > > [...] but TestNG is meant to support functional/integration tests. > If I may requote the TestNG marketing slogan, again: > TestNG is designed to cover all categories of tests: *unit*, [...] TestNG is not designed only for the fancy tests, it can run the simple unit tests, too. Dan, to me it looks you/Surefire are pushing something like "If you want unit tests, use JUnit, if you want function/integration tests, use TestNG" by punishing those that employ TestNG for the simple use-case. I am neither a JUnit nor TestNG fanatic but I can't follow your XOR-like argument here. Benjamin Bentmann -- View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15136338.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
Benjamin Bentmann wrote: > But what I really want is to avoid bothering with suites at all. Sure, > TestNG has a powerful suite concept, nice, maybe we may want to use it > someday in the future but for now, we have just the simple use-case with > non-related test classes, i.e. we are happy by grouping at the class > level. As I said, I don't think you're using the right tool for the job here. TestNG is meant to support all that fancy stuff, and behaves a little differently for that reason. You can get the exact behavior you want from JUnit. > We have no suite(s), only a directory with some unrelated test classes. > Now Surefire goes ahead and creates a suite, automatically, without my > control, assuming the entire directory is a single suite. This > assumption does not fit our needs. That's fair, but I claim that this assumption is a good one for people who are using TestNG, most of whom pick TestNG over JUnit because they need to do that weird stuff. Again, if you want class-level logging, I encourage you to take up the matter with the TestNG team. TestNG isn't meant to work this way today. > dfabulich wrote: >> >> That option sounds really confusing. What would you call it? How would >> you document it? You'd have to know a lot about the guts of Surefire and >> TestNG for an option like that to make much sense. >> > Could think of something like "suiteMode=class|package|all": > This option controls how Surefire will assemble your test classes into a > suite. "class" means to run each class individually in its own suite. > "package" groups all classes from the same package into a separate suite. > "all" will aggregate all found test classes in a single suite. Each created > suite will then be passed to the testing framework for execution and will be > run independently of the other suites. Note that test classes that have > dependencies on each other, must be executed in the same suite. Therefore, > the values "class" and "package" should be used with care because Surefire > itself cannot detect such inter-class dependencies. Doesn't this option smell bad to you? It does to me! Instead of making this option, wouldn't it be nice if it just logged correctly? If TestNG notified us when to log, we could log anything we wanted. Surefire is meant to be a wrapper around the underlying test framework; it's not meant to provide features that the frameworks lack. I think this is one of those cases where trying to provide the feature in Surefire would be way harder, and would also be fighting the framework. > Say one starts of with JUnit, where Surefire executes classes > individually. Then one goes and converts his tests to use TestNG (they > have a tool for this). Conceptually, you have the same test design, but > now Surefire insists on running all classes in a single job/black-box. I > am sorry, but I still find this odd since I am lacking the required > knowledge about the guts of Surefire and TestNG ;-) For the record, I agree that the difference between TestNG and JUnit is strange. I claim that we only find it odd BECAUSE we have painfully learned about these guts... most people don't notice or care about the difference. -Dan
Jason van Zyl wrote: > Correct me if I'm wrong here Benjamin but I think he's talking about > stating what tests you want to run, and how they are grouped > declaratively. As a best practice don't use suites and come up with a > way in Surefire to group things and avoid grouping of things in code. Inter-class dependencies are one of TestNG's fundamental features; it's arguably the most important architectural difference between TestNG and JUnit. As Benjamin pointed out, they are declarative. > I think it would just be a practice for class to be self-contained, test > one thing, and allow wiring up of sets/groups externally. That's a good practice for unit tests, but TestNG is meant to support functional/integration tests. Unlike unit tests, which are meant to be isolated and test one thing, TestNG tests are supposed to test more than one thing together, and are therefore designed to inter-depend on one another. It's a pretty significant philosophical shift; the TestNG guys wrote a whole book about it, blah blah blah. :-) Dependent test methods http://beust.com/weblog/archives/000171.html Are dependent test methods really evil? http://beust.com/weblog/archives/000259.html Why unit tests are disappearing http://beust.com/weblog/archives/000319.html > If this model breaks down then maybe, as I've asked before, it makes sense to > create separate plugins for the various test tools. I'm still on the fence about that, but regardless I don't think this is a good candidate reason. Surefire has to handle this problem in the case of JUnit suites as well as TestNG suites; the only difference is that in JUnit this problem is less important because we think people mostly shouldn't write unit tests like that. -Dan
On 27-Jan-08, at 12:31 PM, Benjamin Bentmann wrote: > > > Jason van Zyl-2 wrote: >> >> Correct me if I'm wrong here Benjamin but I think he's talking about >> stating what tests you want to run, and how they are grouped >> declaratively. As a best practice don't use suites and come up with a >> way in Surefire to group things and avoid grouping of things in code. >> > > Not quite sure if I understand you correctly. A TestNG suite XML > file is > quite declarative so I already have the possibility to declaratively > select > and group tests. In fear that I can't get Surefire where I wished > (one-class-per-suite), I opened http://jira.codehaus.org/browse/SUREFIRE-438 > which would allow to easily feed a bunch of suite XML files into the > plugin. > > But what I really want is to avoid bothering with suites at all. Sorry if I didn't explain it well but we are in 100% agreement. Make tests simple, no suites, and group declaratively. > Sure, > TestNG has a powerful suite concept, nice, maybe we may want to use it > someday in the future but for now, we have just the simple use-case > with > non-related test classes, i.e. we are happy by grouping at the class > level. > Explicitly maintaining suites, either in code or via XML, would just > be lost > worktime here (although I already thought about a plugin that > automatically > creates single-class suites from the existing test classes and > prepares the > dozens of suite files before the actual Surefire run...) > > We have no suite(s), only a directory with some unrelated test > classes. Now > Surefire goes ahead and creates a suite, automatically, without my > control, > assuming the entire directory is a single suite. This assumption > does not > fit our needs. > > > dfabulich wrote: >> >> That option sounds really confusing. What would you call it? How >> would >> you document it? You'd have to know a lot about the guts of >> Surefire and >> TestNG for an option like that to make much sense. >> > Could think of something like "suiteMode=class|package|all": > This option controls how Surefire will assemble your test classes > into a > suite. "class" means to run each class individually in its own suite. > "package" groups all classes from the same package into a separate > suite. > "all" will aggregate all found test classes in a single suite. Each > created > suite will then be passed to the testing framework for execution and > will be > run independently of the other suites. Note that test classes that > have > dependencies on each other, must be executed in the same suite. > Therefore, > the values "class" and "package" should be used with care because > Surefire > itself cannot detect such inter-class dependencies. > > > > dfabulich wrote: >> >> "TestNG is designed to cover all categories of tests: unit, [...]" >> [...] >> If all you're doing is unit tests, you may prefer to use JUnit 4. :-) >> > > Well, no comments ;-) > > Say one starts of with JUnit, where Surefire executes classes > individually. > Then one goes and converts his tests to use TestNG (they have a tool > for > this). Conceptually, you have the same test design, but now Surefire > insists > on running all classes in a single job/black-box. I am sorry, but I > still > find this odd since I am lacking the required knowledge about the > guts of > Surefire and TestNG ;-) > > Regards, > > > Benjamin Bentmann > -- > View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15123679.html > Sent from the Surefire - Developer mailing list archive at Nabble.com. > Thanks, Jason ---------------------------------------------------------- Jason van Zyl Founder, Apache Maven jason at sonatype dot com ---------------------------------------------------------- Simplex sigillum veri. (Simplicity is the seal of truth.)
Jason van Zyl-2 wrote: > > Correct me if I'm wrong here Benjamin but I think he's talking about > stating what tests you want to run, and how they are grouped > declaratively. As a best practice don't use suites and come up with a > way in Surefire to group things and avoid grouping of things in code. > Not quite sure if I understand you correctly. A TestNG suite XML file is quite declarative so I already have the possibility to declaratively select and group tests. In fear that I can't get Surefire where I wished (one-class-per-suite), I opened http://jira.codehaus.org/browse/SUREFIRE-438 which would allow to easily feed a bunch of suite XML files into the plugin. But what I really want is to avoid bothering with suites at all. Sure, TestNG has a powerful suite concept, nice, maybe we may want to use it someday in the future but for now, we have just the simple use-case with non-related test classes, i.e. we are happy by grouping at the class level. Explicitly maintaining suites, either in code or via XML, would just be lost worktime here (although I already thought about a plugin that automatically creates single-class suites from the existing test classes and prepares the dozens of suite files before the actual Surefire run...) We have no suite(s), only a directory with some unrelated test classes. Now Surefire goes ahead and creates a suite, automatically, without my control, assuming the entire directory is a single suite. This assumption does not fit our needs. dfabulich wrote: > > That option sounds really confusing. What would you call it? How would > you document it? You'd have to know a lot about the guts of Surefire and > TestNG for an option like that to make much sense. > Could think of something like "suiteMode=class|package|all": This option controls how Surefire will assemble your test classes into a suite. "class" means to run each class individually in its own suite. "package" groups all classes from the same package into a separate suite. "all" will aggregate all found test classes in a single suite. Each created suite will then be passed to the testing framework for execution and will be run independently of the other suites. Note that test classes that have dependencies on each other, must be executed in the same suite. Therefore, the values "class" and "package" should be used with care because Surefire itself cannot detect such inter-class dependencies. dfabulich wrote: > > "TestNG is designed to cover all categories of tests: unit, [...]" > [...] > If all you're doing is unit tests, you may prefer to use JUnit 4. :-) > Well, no comments ;-) Say one starts of with JUnit, where Surefire executes classes individually. Then one goes and converts his tests to use TestNG (they have a tool for this). Conceptually, you have the same test design, but now Surefire insists on running all classes in a single job/black-box. I am sorry, but I still find this odd since I am lacking the required knowledge about the guts of Surefire and TestNG ;-) Regards, Benjamin Bentmann -- View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15123679.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
On 25-Jan-08, at 10:59 AM, Dan Fabulich wrote: > Benjamin Bentmann wrote: > >> I see. Now, if the user can provide an option to Surefire, >> indicating that the unit tests do not have inter-class >> dependencies, wouldn't this allow to get back the old behavior? >> I.e. enabling Surefire to pass test classes individually into TestNG? > > That option sounds really confusing. What would you call it? How > would you document it? You'd have to know a lot about the guts of > Surefire and TestNG for an option like that to make much sense. > Correct me if I'm wrong here Benjamin but I think he's talking about stating what tests you want to run, and how they are grouped declaratively. As a best practice don't use suites and come up with a way in Surefire to group things and avoid grouping of things in code. If this model breaks down then maybe, as I've asked before, it makes sense to create separate plugins for the various test tools. > Not to mention that it'd be error prone. I think it would just be a practice for class to be self-contained, test one thing, and allow wiring up of sets/groups externally. > Developer X would assess the tests, verifying that there are no > inter-class dependencies, and turn on the flag. Later, developer Y > could introduce an inter-class dependency and go to run the tests, > but find that it's not working. It'd be hard to figure out why > without knowing a lot about how Surefire is doing its job. > > More generally, I don't think class-level reporting really makes > sense in TestNG as it's designed today. Right now, TestNG provides > listeners with the following notifications: > > onStart > onTestStart > onTestSuccess > onTestFailure > onTestSkipped > onTestFailedButWithinSuccessPercentage > onConfigurationSuccess > onConfigurationFailure > onConfigurationSkip > onFinish > > Note that there's nothing here about "onClassStart/Finish" or even > "onGroupStart/Finish". > > If TestNG exposed a listener, we could easily attach to it, but I > see that as a feature of TestNG. We could certainly try to write > TestNG features in Surefire, but the TestNG code knows best whether, > for example, your classes have any inter-dependencies, when your > classes start and stop, and more generally how to construct a full > test run comprised of suites, groups, classes and tests. > > Surefire shouldn't be in the business of reimplementing that logic, > especially since we expect this feature to be generally useful. It > should be a part of TestNG, if it matters at all. > >> I would not have any problems with a single file per method. >> Indeed, I think >> I like the idea: Test methods are the units that succeed or fail, >> so if test >> method TestClass.testFoo() fails, I would investigate >> TestClass.testFoo-output.txt and can easily concentrate on its output >> without any need to skip over the output from all the other >> successful >> methods. I think that would be worth another option ;-) > > "TestNG is designed to cover all categories of tests: unit, > functional, end-to-end, integration, etc..." > > In TestNG, especially when one method @dependsOn another method, the > test failures aren't units, and aren't even meant to be units. > (This can be a huge benefit for slow integration tests, where > earlier tests leave behind fixtures that later tests are meant to > consume.) > > If all you're doing is unit tests, you may prefer to use JUnit 4. :-) > > Anyway, in thinking about it a little harder I realized that we > can't even necessarily split the stdout into one file per method, > since the tests may be run in parallel! > > I'm pretty sure we can do that logging thing, at least. :-p > > -Dan Thanks, Jason ---------------------------------------------------------- Jason van Zyl Founder, Apache Maven jason at sonatype dot com ---------------------------------------------------------- We know what we are, but know not what we may be. -- Shakespeare
It would seem to be best in Surefire to me - the assertion in Maven should really be covered by a unit test? On 26/01/2008, at 5:25 AM, Dan Fabulich wrote: > > Class path ordering is tricky, and changed recently. Benjamin > Bentmann provides a regression test in SUREFIRE-428. > > However, I'm not certain if I should apply it in Surefire, because > we use those integration tests to verify backwards compatibility > with older versions of Maven. > > My initial thought was that perhaps the test should be applied in > core-integration-testing; there are a few small Surefire integration > tests there (which I think is correct), and I think this could join > them to prevent regression for MNG-3118. > > Then again, since it's fundamentally an integration test, it doesn't > specifically belong to either side, so maybe Surefire is the best > place to install it...? > > Thoughts? > > -Dan
On Jan 25, 2008 5:04 PM, Dan Fabulich <dan@fabulich.com> wrote: > > Steve Loughran wrote: > > >> What I propose is that, in order to avoid destroying information, Surefire > >> should generate XML that looks like Example 7 (all-in-one-file), and not > >> try to fake it to look like Example 2 (one-file-per-class). (TestNG's > >> junit-like output also generates files like Example 7.) However, when it > >> comes time to generate an HTML report, surefire-reports will discard > >> suite-level information, and treat large suites like Example 7 as if they > >> had been presented in separate files like Example 2. I'd argue that all > >> of the other JunitReport-like tools (including Ant) should probably follow > >> the same lead. > > > > I think that if everyone else has a bug, its hard to call it a bug. More > > a presentation choice :) > > I'd thought somebody might say that! :-) Still, do you think TestNG and > other tools should therefore generate multiple XML files, to be compatible > with the other (arguably broken) reporting tools? No, * junit report will take other sets of XSL sheets than those built in to Ant's own JAR. This allows anyone to fix the XSLs without waiting for ant's (fairly mature) release cycle. * CI tools are a separate issue. Don't know there Looking at the reports -and trying to imagine a time where I ever get to stop making meta-RPM-RPMs (don't ask), I do want to push the test formats forward in a way everyone is happy. I really dont like the way the current stuff sticks summary info as toplevel attributes (time, results), as it stops us streaming out the XML as we go. I'm starting to wonder if we couldnt move to a new format, and have the option of XSL generation of the legacy stuff, purely for those tools that havent been upgraded in sync. -steve
Class path ordering is tricky, and changed recently. Benjamin Bentmann provides a regression test in SUREFIRE-428. However, I'm not certain if I should apply it in Surefire, because we use those integration tests to verify backwards compatibility with older versions of Maven. My initial thought was that perhaps the test should be applied in core-integration-testing; there are a few small Surefire integration tests there (which I think is correct), and I think this could join them to prevent regression for MNG-3118. Then again, since it's fundamentally an integration test, it doesn't specifically belong to either side, so maybe Surefire is the best place to install it...? Thoughts? -Dan
Benjamin Bentmann wrote: > I see. Now, if the user can provide an option to Surefire, indicating > that the unit tests do not have inter-class dependencies, wouldn't this > allow to get back the old behavior? I.e. enabling Surefire to pass test > classes individually into TestNG? That option sounds really confusing. What would you call it? How would you document it? You'd have to know a lot about the guts of Surefire and TestNG for an option like that to make much sense. Not to mention that it'd be error prone. Developer X would assess the tests, verifying that there are no inter-class dependencies, and turn on the flag. Later, developer Y could introduce an inter-class dependency and go to run the tests, but find that it's not working. It'd be hard to figure out why without knowing a lot about how Surefire is doing its job. More generally, I don't think class-level reporting really makes sense in TestNG as it's designed today. Right now, TestNG provides listeners with the following notifications: onStart onTestStart onTestSuccess onTestFailure onTestSkipped onTestFailedButWithinSuccessPercentage onConfigurationSuccess onConfigurationFailure onConfigurationSkip onFinish Note that there's nothing here about "onClassStart/Finish" or even "onGroupStart/Finish". If TestNG exposed a listener, we could easily attach to it, but I see that as a feature of TestNG. We could certainly try to write TestNG features in Surefire, but the TestNG code knows best whether, for example, your classes have any inter-dependencies, when your classes start and stop, and more generally how to construct a full test run comprised of suites, groups, classes and tests. Surefire shouldn't be in the business of reimplementing that logic, especially since we expect this feature to be generally useful. It should be a part of TestNG, if it matters at all. > I would not have any problems with a single file per method. Indeed, I think > I like the idea: Test methods are the units that succeed or fail, so if test > method TestClass.testFoo() fails, I would investigate > TestClass.testFoo-output.txt and can easily concentrate on its output > without any need to skip over the output from all the other successful > methods. I think that would be worth another option ;-) "TestNG is designed to cover all categories of tests: unit, functional, end-to-end, integration, etc..." In TestNG, especially when one method @dependsOn another method, the test failures aren't units, and aren't even meant to be units. (This can be a huge benefit for slow integration tests, where earlier tests leave behind fixtures that later tests are meant to consume.) If all you're doing is unit tests, you may prefer to use JUnit 4. :-) Anyway, in thinking about it a little harder I realized that we can't even necessarily split the stdout into one file per method, since the tests may be run in parallel! I'm pretty sure we can do that logging thing, at least. :-p -Dan
dfabulich wrote: > > Typically it's used for slow tests; you can identify a subset of tests > that you want to run and just run those. (You can even pull just one > method from Class X, and another method from Class Y, and so on.) > Thanks for the explanation. To me, this seems more useful for ad-hoc testing. During an automated build, I want all existing tests for maximum test coverage of the artifact. dfabulich wrote: > > For TestNG, we have to hand it the entire directory, because any one class > may @dependsOn methods in other classes. > I see. Now, if the user can provide an option to Surefire, indicating that the unit tests do not have inter-class dependencies, wouldn't this allow to get back the old behavior? I.e. enabling Surefire to pass test classes individually into TestNG? dfabulich wrote: > > As for splitting them all up into separate txt files, the only way we > could do it would be to split the tests up into one txt file per METHOD, > which is way too many files IMO. > I would not have any problems with a single file per method. Indeed, I think I like the idea: Test methods are the units that succeed or fail, so if test method TestClass.testFoo() fails, I would investigate TestClass.testFoo-output.txt and can easily concentrate on its output without any need to skip over the output from all the other successful methods. I think that would be worth another option ;-) dfabulich wrote: > > It's odd that you describe this as a 2.3.1 regression... I couldn't get > TestNG to work at all in Surefire 2.3.x, hence my work on Surefire 2.4. > Using Surefire 2.3.x together with TestNG 5.1 worked for me. Nevertheless, it's good to have Surefire 2.4 now! Regards, Benjamin Bentmann -- View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15093881.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
Steve Loughran wrote: >> What I propose is that, in order to avoid destroying information, Surefire >> should generate XML that looks like Example 7 (all-in-one-file), and not >> try to fake it to look like Example 2 (one-file-per-class). (TestNG's >> junit-like output also generates files like Example 7.) However, when it >> comes time to generate an HTML report, surefire-reports will discard >> suite-level information, and treat large suites like Example 7 as if they >> had been presented in separate files like Example 2. I'd argue that all >> of the other JunitReport-like tools (including Ant) should probably follow >> the same lead. > > I think that if everyone else has a bug, its hard to call it a bug. More > a presentation choice :) I'd thought somebody might say that! :-) Still, do you think TestNG and other tools should therefore generate multiple XML files, to be compatible with the other (arguably broken) reporting tools? -Dan
Benjamin Bentmann wrote: > For my curiosity: What would be the benefit of setting up a hand-crafted > test suite? I am a lazy guy and prefer the dumb machine to do the nasty > things for me so I really like the idea of just dropping a test class into > src/test/java without bothering to additionally maintain some test suite, > too. Typically it's used for slow tests; you can identify a subset of tests that you want to run and just run those. (You can even pull just one method from Class X, and another method from Class Y, and so on.) > My personal concerns are > a) the console output from Surefire during test execution and > b) the redirected test output files (*-output.txt) Surefire's design is to hand off control to the testing framework (JUnit/TestNG) and then report on the console when we get control back. For JUnit, we hand off control on a "class" level, so we get control back after every class. For TestNG, we have to hand it the entire directory, because any one class may @dependsOn methods in other classes. Furthermore, TestNG doesn't notify us at the start/end of every class; indeed, it's easy to configure TestNG so that individual class methods run out of order (so that some of Class X runs and then some of Class Y, and then back to Class X, then Class Y finishes, etc.) and that's without considering parallelized testing. We could log to the console on individual test methods as they run, though I'm sure we'd want to make it a configurable option. I've filed SUREFIRE-437 for the console logging issue. As for splitting them all up into separate txt files, the only way we could do it would be to split the tests up into one txt file per METHOD, which is way too many files IMO. > With Surefire 2.3.1, I got a result feedback every half minute when a > test class finished. It's odd that you describe this as a 2.3.1 regression... I couldn't get TestNG to work at all in Surefire 2.3.x, hence my work on Surefire 2.4. -Dan
On Jan 24, 2008 10:12 PM, Dan Fabulich <dan@fabulich.com> wrote: > > Most of the JUnit reporting tools, including Ant's JunitReport task and > Maven's surefire-report plugin (but also including Hudson and others) > incorrectly assume that all of those testcases are methods of MyTestSuite. > As a result, it makes it look like all of your tests are methods of one > class in one package. If you've got two "testOutput" methods, they'll be > indistinguishable (except by stacktrace, if one is present). > > The same thing happens when you run TestNG tests: it always generates all > of the test results in one XML file. (Though it contains more > information, see below.) > > More generally, both Ant's JunitReport and surefire-report incorrectly > assume a one-to-one relationship between TestCase classes, XML files, and > "suites" of tests. They assume that every XML file contains one suite of > tests, which is the same thing as a TestCase class, and all of the tests > in the suite are just methods of the TestCase class. This assumption is > wrong. I guess the reason for the assumption is that nobody normally writes a test suite for JUnit, as it is needless coding that a pattern like **/*Test.class could do at the build file level, where you can also turn different tests on and off. But as you point out, it doesnt hold for other test systems. > Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase> > classes to include the class name of the class on every <testcase> > element: > > Example 7 > <testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031"> > <properties><!-- ... --></properties > <testcase name="test1a" time="0.0" classname="com.mycompany.Test1" /> > <testcase name="test1b" time="0.0" classname="com.mycompany.Test1" /> > <testcase name="test2a" time="0.0" classname="com.mycompany.Test2" /> > <testcase name="test2b" time="0.0" classname="com.mycompany.Test2" /> > <testcase name="test3a" time="0.0" classname="com.mycompany.Test3" /> > <testcase name="test3b" time="0.0" classname="com.mycompany.Test3" /> > </testsuite> I forget when, but take your word for it. > This is good, because it allows you to figure out which class the method > really belongs to. This is what TestNG outputs when it generates > JUnit-like results. Unfortunately, nobody honors the "classname" > attribute, including Ant 1.7's JunitReport task! You contributions to the XSL file are welcome :) > > Even if we did try to honor that information and report on it in the HTML, > how would we do it? Right now, the reports are organized in terms of > packages, and within the packages you'll get a list of classes (assumed > incorrectly to be the same thing as "suites"), and within the list of > classes you'll get a list of test method names. There's no room for > "suites" (as distinct from classes) in these reports at all. > > And arguably there shouldn't be a place for this information, because > apparently 80-90% of the time suites aren't being used; adding a "suites" > section would be redundant in those cases. > > We could throw away the "suite" wrapper and pretend as if the tests were > just classes and methods, but note something else: the sum of the "time" > attribute on the <testcase> elements is normally different from the "time" > attribute on the <testsuite>. That's because the timer has a 16ms > resolution on most operating systems. > > In bug SUREFIRE-433, it was argued that we should generate separate JUnit > XML files for every class. You could even imagine converting the results > from Example 7 into results that looked like Example 2, throwing away all > of the suite-level information. > > I certainly want to be conservative in what I emit and liberal in what I > accept, but I *really* don't want to throw away information, especially at > the XML level. With that said, as we're trying to make the report look > something like the way it looks today, we're going to lose suite-level > information in the HTML. > > What I propose is that, in order to avoid destroying information, Surefire > should generate XML that looks like Example 7 (all-in-one-file), and not > try to fake it to look like Example 2 (one-file-per-class). (TestNG's > junit-like output also generates files like Example 7.) However, when it > comes time to generate an HTML report, surefire-reports will discard > suite-level information, and treat large suites like Example 7 as if they > had been presented in separate files like Example 2. I'd argue that all > of the other JunitReport-like tools (including Ant) should probably follow > the same lead. I think that if everyone else has a bug, its hard to call it a bug. More a presentation choice :) > My proposal is not obviously right, because, again, most other JUnit > report tools also have this reporting bug; when they try to format TestNG > output or Surefire output, they'll incorrectly report all methods to be > members of the suite class. Maybe since we expect EVERYONE to have this > bug, we should be even more conservative in what we emit and generate > multi-file output, just so the other tools will know how to interpret it > correctly. That would make it more likely that the HTML output would > contain complete information. > > What do other people think? Agree, disagree? I do need to allocate time to fix reporting in Ant, both what we generate (assuming a junit4 task), and what the tool processes. Me, I'm more concerned with the problem of trying to merge log results from a server and a client, as the current reports only pick the stdout and stderr from one side of the party, -steve
dfabulich wrote: > > First, it seems like most people don't even use test suites, preferring to > simply run every test in their src/test/java directory. > For my curiosity: What would be the benefit of setting up a hand-crafted test suite? I am a lazy guy and prefer the dumb machine to do the nasty things for me so I really like the idea of just dropping a test class into src/test/java without bothering to additionally maintain some test suite, too. dfabulich wrote: > > What I propose is that, in order to avoid destroying information, Surefire > should generate XML that looks like Example 7 (all-in-one-file), and not > try to fake it to look like Example 2 (one-file-per-class). > At least for me, I do not care about the structure/contents of the XML report files (never read nor parse them by myself). Having a central file with all information sounds like a reasonable approach for me, no problems so far. My personal concerns are a) the console output from Surefire during test execution and b) the redirected test output files (*-output.txt) I am the kind of guy who prefers washing machines with a window, so I really dislike the situation with Surefire 2.4 and TestNG where I see *nothing* on the console until the suite completes (which takes roughly 5 minutes in one of our modules). It is in general no good ui design if the user doesn't get any feedback for long running tasks. With Surefire 2.3.1, I got a result feedback every half minute when a test class finished. Our tests use log4j with debug level logging on the console to track the test execution and to evaluate it in case of failures. Therefore, we have <redirectTestOutputToFile>true</redirectTestOutputToFile>. Now image a test suite with say n test classes, every class producing X KB output. If test class A fails, we go inspect and A-output.txt with an acceptable size of X KB. This becomes less pleasant when you have to inspect the all-in-one TestSuite-output.txt which has a total size of n * X KB and n being a growing factor as the test suite is extended. I hope these illustrations help to understand the intended use-case. Regards, Benjamin Bentmann -- View this message in context: http://www.nabble.com/Test-Suites%2C-Ant%2C-Surefire%2C-and-JunitReport-tp15076378p15084183.html Sent from the Surefire - Developer mailing list archive at Nabble.com.
I spent some time solving a thorny problem over the last couple of days;
I'm not 100% sure I like the solution to it, so I wanted to write a bit
about it to see what other people think.
The problem occurs in cases where one test suite incorporates tests from
multiple test classes. This is the normal use case for a JUnit TestSuite.
You'll typically write a JUnit suite like this:
Example 1
public class MyTestSuite extends TestSuite {
public static Test suite() {
MyTestSuite suite = new MyTestSuite();
suite.addTestSuite(FooTestCase.class);
suite.addTest(BarTestSuite.suite());
return suite;
}
}
(In TestNG, suites are defined using XML files, and are considerably more
common in TestNG-land.)
The question I'm trying to answer is: how should the results of running a
TestSuite be represented in XML, and how should that XML be represented in
HTML? You might think this would be a solved problem, but it doesn't
appear to be. :-( Bugs SUREFIRE-433 and TESTNG-204 all relate to this, as
well as Ant bug 24106.
http://jira.codehaus.org/browse/SUREFIRE-433
http://jira.opensymphony.com/browse/TESTNG-204
http://issues.apache.org/bugzilla/show_bug.cgi?id=24106
First, it seems like most people don't even use test suites, preferring to
simply run every test in their src/test/java directory. If you do this,
you get a bunch of XML files, one file per TestCase class, like this:
Example 2
TEST-com.mycompany.Test1.xml
TEST-com.mycompany.Test2.xml
TEST-com.mycompany.Test3.xml
The individual XML files look like this:
Example 3
<testsuite name="com.mycompany.Test1" failures="1" errors="0" tests="2" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0">
<failure type="java.lang.AssertionError" message="I don't like this movie!">java.lang.AssertionError:
I don't like this movie!
at org.testng.Assert.fail(Assert.java:84)
at org.testng.Assert.fail(Assert.java:91)
at com.mycompany.Test1.test1b(Test1.java:14)
</failure>
</testcase>
</testsuite>
That's all fine, but what if you want to use a test suite? When you do
that in Ant or in Maven today, you get just one XML file:
Example 4
TEST-com.mycompany.MyTestSuite.xml
In Surefire 2.4, the XML unfortunately looks like this:
Example 5
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0" />
<testcase name="test2a" time="0.0" />
<testcase name="test2b" time="0.0" />
<testcase name="test3a" time="0.0" />
<testcase name="test3b" time="0.0" />
</testsuite>
Note that information has been lost here, and the information that *is*
provided is misleading. Specifically, we lost the fact that test1a was a
method of com.mycompany.Test1; all we know is its method name.
Furthermore, the report misleadingly seems to suggest that test1a is a
method of MyTestSuite, whereas in fact MyTestSuite just has a suite()
method and no test methods of its own.
Most of the JUnit reporting tools, including Ant's JunitReport task and
Maven's surefire-report plugin (but also including Hudson and others)
incorrectly assume that all of those testcases are methods of MyTestSuite.
As a result, it makes it look like all of your tests are methods of one
class in one package. If you've got two "testOutput" methods, they'll be
indistinguishable (except by stacktrace, if one is present).
The same thing happens when you run TestNG tests: it always generates all
of the test results in one XML file. (Though it contains more
information, see below.)
More generally, both Ant's JunitReport and surefire-report incorrectly
assume a one-to-one relationship between TestCase classes, XML files, and
"suites" of tests. They assume that every XML file contains one suite of
tests, which is the same thing as a TestCase class, and all of the tests
in the suite are just methods of the TestCase class. This assumption is
wrong.
Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase>
classes to include the class name of the class on every <testcase>
element:
Example 7
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test1b" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test2a" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test2b" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test3a" time="0.0" classname="com.mycompany.Test3" />
<testcase name="test3b" time="0.0" classname="com.mycompany.Test3" />
</testsuite>
This is good, because it allows you to figure out which class the method
really belongs to. This is what TestNG outputs when it generates
JUnit-like results. Unfortunately, nobody honors the "classname"
attribute, including Ant 1.7's JunitReport task!
Even if we did try to honor that information and report on it in the HTML,
how would we do it? Right now, the reports are organized in terms of
packages, and within the packages you'll get a list of classes (assumed
incorrectly to be the same thing as "suites"), and within the list of
classes you'll get a list of test method names. There's no room for
"suites" (as distinct from classes) in these reports at all.
And arguably there shouldn't be a place for this information, because
apparently 80-90% of the time suites aren't being used; adding a "suites"
section would be redundant in those cases.
We could throw away the "suite" wrapper and pretend as if the tests were
just classes and methods, but note something else: the sum of the "time"
attribute on the <testcase> elements is normally different from the "time"
attribute on the <testsuite>. That's because the timer has a 16ms
resolution on most operating systems. By starting the timer at the
beginning of the entire suite, we can capture overall time that we lose
when we turn the timer on and off. (And what if the tests had been run in
parallel?)
In bug SUREFIRE-433, it was argued that we should generate separate JUnit
XML files for every class. You could even imagine converting the results
from Example 7 into results that looked like Example 2, throwing away all
of the suite-level information.
I certainly want to be conservative in what I emit and liberal in what I
accept, but I *really* don't want to throw away information, especially at
the XML level. With that said, as we're trying to make the report look
something like the way it looks today, we're going to lose suite-level
information in the HTML.
What I propose is that, in order to avoid destroying information, Surefire
should generate XML that looks like Example 7 (all-in-one-file), and not
try to fake it to look like Example 2 (one-file-per-class). (TestNG's
junit-like output also generates files like Example 7.) However, when it
comes time to generate an HTML report, surefire-reports will discard
suite-level information, and treat large suites like Example 7 as if they
had been presented in separate files like Example 2. I'd argue that all
of the other JunitReport-like tools (including Ant) should probably follow
the same lead.
My proposal is not obviously right, because, again, most other JUnit
report tools also have this reporting bug; when they try to format TestNG
output or Surefire output, they'll incorrectly report all methods to be
members of the suite class. Maybe since we expect EVERYONE to have this
bug, we should be even more conservative in what we emit and generate
multi-file output, just so the other tools will know how to interpret it
correctly. That would make it more likely that the HTML output would
contain complete information.
What do other people think? Agree, disagree?
-Dan
Brett Porter wrote: > Hi Dan, > > I'm just flipping back through the commits by way of a quick review and had a > question about this one. > > Does TestNG set a default thread count once parallel is enabled? > > I assume the problematic configuration is parallel = false + any thread > count, so maybe a better alternative is to set the default, and manually > unset it if parallel is false? "-parallel" doesn't mean what you think it means... in fact, this value was confusing TestNG because it's not a boolean (any more). Since 5.5(?) it would be better to call it "-parallelmode"; its values can be either "methods" or "tests" or it can be undefined. It has no effect whatsoever if -threadcount isn't set. Furthermore, -threadcount has no effect unless -parallel is set to something. Does this make sense? No, not really. But I also don't think it would be a good idea for Surefire to get into the business of managing this. In 2.4 we just pass the string along to TestNG and hope for the best. -Dan
Hi Dan,
I'm just flipping back through the commits by way of a quick review
and had a question about this one.
Does TestNG set a default thread count once parallel is enabled?
I assume the problematic configuration is parallel = false + any
thread count, so maybe a better alternative is to set the default, and
manually unset it if parallel is false?
Cheers,
Brett
On 21/11/2007, at 8:58 AM, dfabulich@apache.org wrote:
> Author: dfabulich
> Date: Tue Nov 20 16:58:42 2007
> New Revision: 596894
>
> URL: http://svn.apache.org/viewvc?rev=596894&view=rev
> Log:
> [SUREFIRE-361] Don't set threadcount or parallel by default; these
> defaults confuse TestNG 5.7
>
> Modified:
> maven/surefire/trunk/maven-surefire-plugin/src/main/java/org/
> apache/maven/plugin/surefire/SurefirePlugin.java
>
> Modified: maven/surefire/trunk/maven-surefire-plugin/src/main/java/
> org/apache/maven/plugin/surefire/SurefirePlugin.java
> URL: http://svn.apache.org/viewvc/maven/surefire/trunk/maven-surefire-plugin/src/main/java/org/apache/maven/plugin/surefire/SurefirePlugin.java?rev=596894&r1=596893&r2=596894&view=diff
> =
> =
> =
> =
> =
> =
> =
> =
> ======================================================================
> --- maven/surefire/trunk/maven-surefire-plugin/src/main/java/org/
> apache/maven/plugin/surefire/SurefirePlugin.java (original)
> +++ maven/surefire/trunk/maven-surefire-plugin/src/main/java/org/
> apache/maven/plugin/surefire/SurefirePlugin.java Tue Nov 20 16:58:42
> 2007
> @@ -328,7 +328,7 @@
> * (TestNG only) The attribute thread-count allows you to
> specify how many threads should be allocated for this execution. Only
> * makes sense to use in conjunction with parallel.
> *
> - * @parameter expression="${threadCount}" default-value="5"
> + * @parameter expression="${threadCount}"
> */
> private int threadCount;
>
> @@ -337,7 +337,7 @@
> * methods that depend on each other, which will be run in the
> same thread in order to respect their order of
> * execution.
> *
> - * @parameter expression="${parallel}" default-value="false"
> + * @parameter expression="${parallel}"
> * @todo test how this works with forking, and console/file
> output parallelism
> */
> private String parallel;
>
>
Stephen Connolly wrote: > Can we make the property shorter and easier to remember... > > that's what I liked about the -Dtest=0 hack Just for you, I've filed SUREFIRE-417 (Make new "skipTests" parameter to replace skipExec). ;-) I've got to fix SUREFIRE-416 anyway, and I'd forgotten that this has always bothered me. :-) "mvn -DskipTests" is short enough, right? I thought about making it "skipTest" but I thought it might sound like we were skipping the entire "test" lifecycle phase...? -Dan
Hi, Maven Surefire version 2.4 is on the runway. Hopefully folks have spent some time trying out the SNAPSHOT version, because we expect ordinary users to get auto-upgraded to version 2.4 when we decide to release. This version fixes numerous long-outstanding bugs, notably in TestNG support. We solved 71 issues: http://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=10541&styleName=Html&version=13243 There are still 31 issues left in JIRA: http://jira.codehaus.org/secure/IssueNavigator.jspa?reset=true&pid=10541&status=1 Staging repo: http://people.apache.org/~dfabulich/staging-repo/ Vote open for 72 hours. [ ] +1 [ ] +0 [ ] -1 PS This is my first call for a release. Be gentle. ;-)
On 12/12/2007, at 6:52 AM, Dan Fabulich wrote: > Brett Porter wrote: > >> I'm just catching up again. I am planning to scan through all the >> commits and see if anything catches my eye. I also saw an issue >> reported on the users list that I'll drop into JIRA about the new >> code working with 2.0.6 - which versions of Maven have you been >> testing with? (I've tended to test with 2.0.4 and 2.0.6, but now >> 2.0.7 and 2.0.8 made sometimes funky changes that are worth running >> the integration tests with). > > I've been pretty much exclusively testing with 2.0.7. I just ran > the 2.4 integration tests with 2.0.6 and 2.0.8 with no problems. > > maven-surefire-plugin lists 2.0.6 as a prerequisite, as of revision > 598160. "mkleint: plexus-utils version 1.4.x means the plugins > requires 2.0.6 at least". Not entirely certain why that is, but > there you have it. 2.0.6 was the first version to shade the plexus-utils dependency away from the plugins. > > >> WDYT about continuing with the 2.3.1 release immediately, and >> giving 2.4 a short time more to simmer? If for any reason 2.4 does >> break certain builds, there is still a better-than-2.3 fallback. >> You can also use the release ann. to re-encourage testing of 2.4. > > +0 on letting 2.4 simmer; -0 on releasing 2.3.1 now. I think I > wouldn't want to let 2.4 simmer for more than two weeks, at the > maximum. > > What concerns me about 2.3.1 is that it doesn't have integration > tests like 2.4, so it's hard to know for sure whether it really *is* > better than 2.3 or whether new bugs have creeped in. (Surefire has > a lot of corner cases and gotchas.) That was kind of the intent of 2.3.1 - limited fixes for known regressions and small issues in 2.3... at least with the work I did on it I felt pretty confident. I see your point though, and I'm happy either way, just thought I'd throw the suggestion out there. Cheers, Brett
Brett Porter wrote: > I'm just catching up again. I am planning to scan through all the commits and > see if anything catches my eye. I also saw an issue reported on the users > list that I'll drop into JIRA about the new code working with 2.0.6 - which > versions of Maven have you been testing with? (I've tended to test with 2.0.4 > and 2.0.6, but now 2.0.7 and 2.0.8 made sometimes funky changes that are > worth running the integration tests with). I've been pretty much exclusively testing with 2.0.7. I just ran the 2.4 integration tests with 2.0.6 and 2.0.8 with no problems. maven-surefire-plugin lists 2.0.6 as a prerequisite, as of revision 598160. "mkleint: plexus-utils version 1.4.x means the plugins requires 2.0.6 at least". Not entirely certain why that is, but there you have it. > WDYT about continuing with the 2.3.1 release immediately, and giving 2.4 > a short time more to simmer? If for any reason 2.4 does break certain > builds, there is still a better-than-2.3 fallback. You can also use the > release ann. to re-encourage testing of 2.4. +0 on letting 2.4 simmer; -0 on releasing 2.3.1 now. I think I wouldn't want to let 2.4 simmer for more than two weeks, at the maximum. What concerns me about 2.3.1 is that it doesn't have integration tests like 2.4, so it's hard to know for sure whether it really *is* better than 2.3 or whether new bugs have creeped in. (Surefire has a lot of corner cases and gotchas.) On the other hand, I feel pretty good about 2.4-SNAPSHOT because it has been getting a lot of incidental testing through users who want/need TestNG support. TestNG recommends using 2.4-SNAPSHOT on their website; pretty much everyone using TestNG and Maven has been using 2.4-SNAPSHOT all along. If Surefire didn't automatically grab the latest release, I'd cheerfully agree to release both and let users sort out what works best for them, but as it stands I'd prefer to destabilize Surefire infrequently and put all of our testing eggs in one basket. With that said, if there's a need for 2.3.1 [Mauro suggested he'd like it last week :-)] then I certainly won't stand in its way. I'd strongly suggest that whoever wanted to try this should run the 2.4 integration tests against 2.3 and then again against 2.3.1 to identify regressions. Seems like a lot of trouble for not much benefit, IMO. -Dan
Guys, This is the issue I was referring to - looks like just the classpath ordering, and so can just be documented. Or is that overlooking a potential issue with 2.3.1 on 2.0.6? Cheers, Brett ---------- Forwarded message ---------- From: Ronn.Chinowutthichai@toyota.com.au <Ronn.Chinowutthichai@toyota.com.au> Date: 10 Dec 2007 16:45 Subject: Re: Backwards incompatibility with 2.0.8? To: Maven Users List <users@maven.apache.org> Cc: Maven Users List <users@maven.apache.org> I've worked out what happened. Surefire 2.3.1 is not in the central repository yet but apparantly it has been tagged (http://jira.codehaus.org/browse/MNG-3118). Last Friday, I've downloaded maven source and build it on my local machine (and added apache snapshot into Artifactory). As a result, surefire 2.3.1-SNAPSHOT got into our Artifactory. This is what I've found. Maven 2.0.6 + surefire 2.3 works. Maven 2.0.6 + surefire 2.3.1-SNAPSHOT doesn't work. Maven 2.0.8 + surefire 2.3 doesn't work Maven 2.0.8 + surefire 2.3.1-SNAPSHOT works. I'm not sure what differences there are in surefire 2.3.1-SNAPSHOT and 2.3.1. I can't find surefire 2.3.1 anywhere so I'll wait and see. Thanks for your help. Cheers, rOnn c. "Brett Porter" <brett.porter@gmail.com> 12/10/2007 11:24 AM Please respond to "Maven Users List" <users@maven.apache.org> To "Maven Users List" <users@maven.apache.org> cc Subject Re: Backwards incompatibility with 2.0.8? Your act of upgrading certainly can't have broken his environment :) There are two possibilities: a) something changed in a surefire snapshot. This is possible as its under active development, though I'm not sure if new snapshots are being deployed regularly. b) a test was added to your environment that relied on the classpath ordering. b) still sounds like the most likely based on your original message - note that 2.0.6 always had main-first ordering, while 2.0.8 has test-first ordering. If you have a sample case where 2.0.6 + surefire 2.3 succeeds and 2.0.6 + 2.3.1 fails, please file it in the surefire JIRA. Otherwise, I would look at the test cases as above. Note you can also use the enforcer to force a Maven version to require people to upgrade to 2.0.8 - consistency is likely to give you less headaches, though I can understand that this is not something you would want to jump to quite so soon perhaps given it is a recent release. I'm still using 2.0.7 myself :) Cheers, Brett On 10/12/2007, Ronn.Chinowutthichai@toyota.com.au <Ronn.Chinowutthichai@toyota.com.au> wrote: > Ok, but our problem is not a simple backward compatibility. > > I was working on project A on my machine. I upgraded to 2.0.8. Nothing > breaks here but it it break then fair enough. > > A colleauge was working on project B and still using 2.0.6. > > Some how my action of upgrading to 2.0.8 in my machine has cause his build > environment to break in his machine. > > Both build machine uses surefire 2.3.1-SNAPSHOT. So perhaps surefure 2.3.1 > doesn't work with Maven 2.0.6 ? > > > rOnn c. > > > > > > > "Brett Porter" <brett.porter@gmail.com> > 12/10/2007 10:42 AM > Please respond to > "Maven Users List" <users@maven.apache.org> > > > To > "Maven Users List" <users@maven.apache.org> > cc > > Subject > Re: Backwards incompatibility with 2.0.8? > > > > > > > No, it's because 2.0.8 is not backwards compatible with 2.0.6, as > documented in the release notes. > > On 10/12/2007, Michael McCallum <gholam@apache.org> wrote: > > its just coincidental that surefire 2.3.1 was released at the same time > as > > 2.0.8... > > > > if you specify surefire 2.3 or less in you pluginManagement then the > tests > > will start working again... of course its best practice to specify > plugin > > versions anyway so magic upgrades don't break things > > > > On Mon, 10 Dec 2007 12:30:25 Ronn.Chinowutthichai@toyota.com.au wrote: > > > We were using maven 2.0.6 in a team with Artifactory as a proxy. > > > > > > I then upgrade my environment to 2.0.8. A whole bunch of new plugins > get > > > fetched. > > > > > > Other people were using 2.0.6 but it appears that they also get a > whole > > > heap of new plugins. > > > > > > As a result, test cases in another project which had worked before now > > > failed for people running 2.0.6. It appears that the problem is with > > > classpath ordering - for some reason with 2.0.6 and the new bunch of > > > plugins, src/main/resources gets picked up before src/test/resources > and > > > so it doesn't load the resources for test cases. > > > > > > So what happens now is that everyone has to abandon 2.0.6 to 2.0.8. > > > > > > Can someone offer any explanation how this could happen? > > > > > > It concerns me that someone running 2.0.6 would now be forced to move > to > > > 2.0.8 simply because I decided to upgrade to 2.0.8 in my own > environment > > > and project? Is the automatic plugin update a problem here? > > > > > > I don't understand how automatic plugin update works but how would one > > > guarantee repeatable builds ? esp with the releases that have been > tagged > > > and potentailly checked out for build later. > > > > > > > > > rOnn c. > > > ###################################################################### > > > DISCLAIMER: > > > This email and any attachment may contain confidential information. > > > If you are not the intended recipient you are not authorized to copy > > > or disclose all or any part of it without the prior written consent > > > of Toyota. > > > > > > Opinions expressed in this email and any attachments are those of the > > > sender and not necessarily the opinions of Toyota. > > > Please scan this email and any attachment(s) for viruses. > > > Toyota does not accept any responsibility for problems caused by > > > viruses, whether it is Toyota's fault or not. > > > ###################################################################### > > > > > > > > -- > > Michael McCallum > > Enterprise Engineer > > mailto:gholam@apache.org > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org > > For additional commands, e-mail: users-help@maven.apache.org > > > > > > > -- > Brett Porter > Blog: http://www.devzuz.org/blogs/bporter/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org > For additional commands, e-mail: users-help@maven.apache.org > > > > ###################################################################### > DISCLAIMER: > This email and any attachment may contain confidential information. > If you are not the intended recipient you are not authorized to copy > or disclose all or any part of it without the prior written consent > of Toyota. > > Opinions expressed in this email and any attachments are those of the > sender and not necessarily the opinions of Toyota. > Please scan this email and any attachment(s) for viruses. > Toyota does not accept any responsibility for problems caused by > viruses, whether it is Toyota's fault or not. > ###################################################################### > -- Brett Porter Blog: http://www.devzuz.org/blogs/bporter/ --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@maven.apache.org For additional commands, e-mail: users-help@maven.apache.org ###################################################################### DISCLAIMER: This email and any attachment may contain confidential information. If you are not the intended recipient you are not authorized to copy or disclose all or any part of it without the prior written consent of Toyota. Opinions expressed in this email and any attachments are those of the sender and not necessarily the opinions of Toyota. Please scan this email and any attachment(s) for viruses. Toyota does not accept any responsibility for problems caused by viruses, whether it is Toyota's fault or not. ###################################################################### -- Brett Porter Blog: http://www.devzuz.org/blogs/bporter/
I'd go with (1). I expect it will be clear enough why it's failing so people will pick up the new usage fast enough. Can you mention the "failIfNoTests=false" flag in the error message? (I assume the true is a typo below?) (3) is a possibility, but you can't really detect if tests are run at "some point" as you say other than externally gathering the info in surefire by aggregating the results. Sounds like more work than its worth. - Brett On 11/12/2007, at 7:02 AM, Dan Fabulich wrote: > > SUREFIRE-350 suggests that "if test parameter is provided, and no > match is found, an error should occur, not a successful build with 0 > tests." That made sense to me, so I checked in a fix in revision > 597952. > > However, I discovered that this broke a certain maybe-standard > usage: if you've got a parent aggregator POM and two children "foo" > and "bar" running in a reactor, you used to be able to say - > Dtest=BarTest; we'd compile foo and run no tests in foo, then run > BarTest in bar. > > This usage very convenient, because you don't have to know that > BarTest is in the bar module; you can just kick off the tests from > the root module and run them in whatever module happens to contain > them. You could also run multiple individual tests separated by > commas in multiple projects. > > Now, if you try to run that line from the parent, you'll get a build > failure in foo, because BarTest isn't defined in foo. > > I can think of a number of different possible options here. > > 1) Leave the code as is. Just because it's convenient doesn't mean > it's important and we have to preserve it. > > 2) Go back and "unfix" SUREFIRE-350. > > Notably, in order to fix SUREFIRE-350, I added a new plugin > parameter called failIfNoTests, which would fail if no tests were > run. (It seemed imaginable that somebody would want to turn that on > in a variety of cases, not just in the -Dtest case.) > > If I were to unfix SUREFIRE-350 it'd still be possible to get a > build failure, if you wanted one really badly, by saying "mvn test - > Dtest=FooTest -DfailIfNoTests=true". > > It's not pretty, but perhaps it's an acceptable compromise. > > 3) Try to add code that detects the reactor case and handles it. I > think it's not that hard to detect whether a given invocation of > SurefirePlugin is running in a reactor (though I don't actually know > how off-hand). But how could I detect the case that this is meant > to help, where you accidentally had a typo? There's no way to > detect whether a test got run *at some point* in the reactor, is > there? > > -Dan
Thanks Dan! I'm just catching up again. I am planning to scan through all the commits and see if anything catches my eye. I also saw an issue reported on the users list that I'll drop into JIRA about the new code working with 2.0.6 - which versions of Maven have you been testing with? (I've tended to test with 2.0.4 and 2.0.6, but now 2.0.7 and 2.0.8 made sometimes funky changes that are worth running the integration tests with). WDYT about continuing with the 2.3.1 release immediately, and giving 2.4 a short time more to simmer? If for any reason 2.4 does break certain builds, there is still a better-than-2.3 fallback. You can also use the release ann. to re-encourage testing of 2.4. And again, thanks - the quantity of work you did here was just awesome :) - Brett On 11/12/2007, at 10:25 AM, Dan Fabulich wrote: > > Surefire trunk currently depends on a couple of SNAPSHOT versions > that we expect to be released in the next couple of days, but other > than that, as far as I know, there's nothing else standing in the > way of a Surefire 2.4 release. It includes more than 50 bug fixes. > > With that said, the more testing we get for Surefire 2.4 the better, > because (as I understand it) people are going to be auto-upgraded to > Surefire 2.4 when it gets released; when Surefire 2.3 came out, it > wasn't uncommon to hear people complaining that their tests used to > work earlier in the week, but post-2.3 no longer work. > > The easiest way you can help us test Surefire is to pull down > surefire trunk and do "mvn clean install"; that will run all of the > Surefire integration tests. If any of those fail, please file a bug > with a stacktrace (and the log.txt from your surefire-integration- > tests/target directory, if applicable). > > https://svn.apache.org/repos/asf/maven/surefire/trunk > > The second easiest way you can help test Surefire 2.4 is to try the > latest SNAPSHOT out in various projects you support. If you find > that Surefire 2.4-SNAPSHOT breaks anything, file bugs and we'll try > to address them. (Reduced test cases in the form of minimal Maven > projects that reproduce the problem would be greatly appreciated.) > > -Dan