Return-Path: Delivered-To: apmail-incubator-river-dev-archive@minotaur.apache.org Received: (qmail 72334 invoked from network); 28 Apr 2009 13:31:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 28 Apr 2009 13:31:12 -0000 Received: (qmail 59634 invoked by uid 500); 28 Apr 2009 13:31:12 -0000 Delivered-To: apmail-incubator-river-dev-archive@incubator.apache.org Received: (qmail 59601 invoked by uid 500); 28 Apr 2009 13:31:12 -0000 Mailing-List: contact river-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: river-dev@incubator.apache.org Delivered-To: mailing list river-dev@incubator.apache.org Received: (qmail 59591 invoked by uid 99); 28 Apr 2009 13:31:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 13:31:12 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [61.9.168.143] (HELO nskntmtas03p.mx.bigpond.com) (61.9.168.143) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 13:31:02 +0000 Received: from nskntotgx02p.mx.bigpond.com ([61.9.223.241]) by nskntmtas03p.mx.bigpond.com with ESMTP id <20090428133039.ELVT426.nskntmtas03p.mx.bigpond.com@nskntotgx02p.mx.bigpond.com> for ; Tue, 28 Apr 2009 13:30:39 +0000 Received: from [10.1.1.2] (really [61.9.223.241]) by nskntotgx02p.mx.bigpond.com with ESMTP id <20090428133038.ESTI27.nskntotgx02p.mx.bigpond.com@[10.1.1.2]> for ; Tue, 28 Apr 2009 13:30:38 +0000 Message-ID: <49F70553.1050701@zeus.net.au> Date: Tue, 28 Apr 2009 23:32:03 +1000 From: Peter Firmstone User-Agent: Thunderbird 2.0.0.14 (X11/20080531) MIME-Version: 1.0 To: river-dev@incubator.apache.org Subject: Re: JUnit vs. jtreg (Was: Build failed in Hudson: River-trunk #163) References: <510143ac0904240613y2e5595f0k8c6cda1abe108f53@mail.gmail.com> <49F2F8F2.3010308@zeus.net.au> <49F4309E.30108@zeus.net.au> <964EAC824495234A86F3C47DA8BD8AAD177611@sucden-exch.sucden.co.uk> <49F6517C.5060203@zeus.net.au> <759a1cdf0904280405s427fe4c1pf21dee281c9efca9@mail.gmail.com> In-Reply-To: <759a1cdf0904280405s427fe4c1pf21dee281c9efca9@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-RPD-ScanID: Class unknown; VirusThreatLevel unknown, RefID str=0001.0A150201.49F704FF.0053,ss=1,fgs=0 X-Virus-Checked: Checked by ClamAV on apache.org Jonathan Costers wrote: > Hi > > I strongly believe we should go the JUnit way for newly written unit tests. > Since JUnit is the de facto standard, is integrated tightly with Ant and > most IDEs ... I see no reason to not use it. > There appears to be strong support among the current commiters to make the decision to create all new tests with JUnit now, no point swimming against the tide. When you get some time, can you create the test directory as per River-303, I'll submit JUnit test patches against River-272. The Unit test for ClassDep will require a dependant group of class files that have all the features that ClassDep checks for, these class files cannot be existing Java platform classes or River classes, as these are subject to change. I'll probably need some help in identifying some suitable candidate code. Could someone clarify for me if JUnit now applies for all new tests or just new unit tests? > How we will deal with the legacy of tests we currently have is another > issue. > We got the QA suite running under Ant, I'm sure we can do the same for the > jtreg tests. > Have a look at River-304, jtreg can be run very simply from ant. The biggest problem I'm having getting the jtreg tests to pass appears to be a security manager issue "access denied" for the http codebase server and RMI . Best Regards, Peter. My Thoughts: One side effect of this decision is there isn't much motivation now for me to investigate jtreg any further than getting the existing tests running. I personally would have liked to take a deeper look at the existing jtreg tool & tests before making this decision, that however, is probably more a reflection of my background, I'm not a programmer per se, but a Mechanical Engineer. We have a deep respect for the achievements of those who came before and like to make such decisions after developing a good understanding from investigation. The lesson isn't learned from Theory, but purely from practice. This isn't to suggest that making big decisions sooner is wrong, just new to me, perhaps it is more applicable to programming. I suppose you can easily change your mind later? In Engineering it's better to change your mind in the design phase, before expending labour and materials. I had hoped to inspire investigation when posing my original question, my reasoning to delay making a decision was based on the following realisations: * We don't yet understand jtreg's capabilities, what can we learn from it, what does it offer? * Does junit provides full equivalent functionality? * How different is jtreg, from what developers are used to? Is it easy to learn, is there IDE and Ant support? * Initial indications are; we cannot guarantee that replacement tests will perform the original tests function, design or intent. (in the case of new code) * We have an existing code base of tests to maintain. Understanding comes from use, the lack of which may directly affect the quality of our code. If there is no motivation for new developers to understand and learn the existing tests & tools, then I worry they may fall into disrepair. * How much new API code are we writing? Will the new unit tests be a small handful in comparison to hundreds of legacy tests? If so then will this cause more confusion for new developers? * JUnit tests, for River's API; if we write new code, honoring the existing API and if an old jtreg unit test exists, then is the existing unit test preferable to check we're not breaking backward compatibility? Or do we write new Unit tests against the existing River API before writing new code? * Brief investigation of jtreg on the net indicated that there was IDE support (Eclipse and Netbeans) while not the most popular test suite, it is used for the JDK. * I can fully understand immediate support for JUnit if we were merging a River dependent project like Rio that itself depended on junit. > I hope to be able to free up some time soon to try that. > > Best > Jonathan > > 2009/4/28 Peter Firmstone > > >> Here's some background: >> >> I was writing junit tests for the new ClassDep implementation on the >> understanding that River didn't have any existing unit testing. Then >> reading the river-dev conversation between Jonathan and Peter, I discover >> that River has existing unit and regression tests. So I posed the question: >> Should I be writing the unit tests using jtreg instead of junit? >> >> I've decided for the time being that until I feel more comfortable, being >> the person who has just requested a test directory be created for the junit >> tests, that we fully understand the implications of doing this before just >> jumping in. I'm not saying we don't do it, just too much too fast for me >> that's all. >> >> Cheers, >> >> Peter. >> >> >> >> Tom Hobbs wrote: >> >> >>> I'm not proposing to reimplement or remove anything. Lets just put >>> these tests as part of the build, the theory being that they /should/ >>> all pass. >>> >>> Then adding/changing River behaviour should then provoke a detailed >>> critique of any jtregs that might then fail. That critique would then >>> conclude to modify/remove the affected jtreg only and it would be done >>> on a very specific basis. >>> >>> There is no reason why /new/ code and /new/ bugs cannot be covered using >>> a more common (junit?) tool. Of course, this assumes that the community >>> agrees to adopt the more common tool over /extending/ the jtreg >>> coverage. >>> >>> To be honest, I haven't had a chance to read any of the jtreg code yet. >>> I'm just adverse to creating more work (reimplementing them) or creating >>> danger (deleting them). >>> I don't really understand what the discussion about them is, can't we >>> just include them in the build script and see what happens? What am I >>> missing? >>> >>> Why one or the other? Why not both with the understanding the jtreg is >>> historic and junit is new stuff? >>> >>> Cheers, >>> >>> Tom >>> >>> -----Original Message----- >>> From: Peter Firmstone [mailto:jini@zeus.net.au] Sent: 26 April 2009 11:00 >>> To: river-dev@incubator.apache.org >>> Subject: Re: JUnit vs. jtreg (Was: Build failed in Hudson: River-trunk >>> #163) >>> >>> Good to have you back Tom, >>> >>> What I'm afraid of, is an uninformed decision being made, we need to do >>> our homework here, so please dive in and check out the code. Thanks to >>> Jonathan asking the right questions, and someone with the knowledge replying >>> (thank you Peter Jones for responding) we've now found out there are >>> existing Regression and Unit tests that rely on jtreg, in addition to the >>> Integration test kit that is now runnable from an ant target. >>> >>> Regression tests are created every time a bug is identified in River's >>> API, the conditions under which it occurs are recreated in one or more >>> regression tests. Later when that part of the API is reimplemented, >>> (assuming also that we're not deliberately breaking backward compatibility) >>> the regression tests confirm that a previously identified >>> >>> bug hasn't been reintroduced. This is important, some of the finer >>> implementation details may not be apparent to the future developer, the >>> regression test allows us to ensure the bug doesn't get reintroduced at a >>> later date. This is obviously very important for security bugs too. >>> >>> A problem with reimplementing the regression tests with another tool is >>> that we would need the old code that included the bug to confirm that the >>> new regression test implementation does in fact reproduce the bug, otherwise >>> our efforts would be for naught. >>> >>> I have a feeling that if we decide to no longer use jtreg for unit tests >>> >>> that we may still need to use it for Regression testing, however I'd like >>> to reserve any further judgment on my part until my understanding improves >>> further. >>> >>> Tom hobbs wrote: >>> >>> >>> >>>> Hi guys, >>>> >>>> Sorry for my recent silence. >>>> >>>> These jtreg doo-hickeys sound like the QA tests to me. They're something >>>> we're inherited and not really sure what to do with them, so >>>> >>>> >>>> >>> >>> >>>> I suggest we treat them like the QA tests. >>>> >>>> Assuming it's not too difficult, lets get running the jtregs as part of >>>> the build. As River functionality is added/changed and individual jtregs >>>> start breaking whoever made the change should assume that it's their >>>> responsibility to modify or remove the jtreg (and QA test) as appropriate. >>>> >>>> Meanwhile, new code gets junit tested. >>>> >>>> Thoughts? >>>> >>>> Also, +1 Java 5. (Java 6 I can take or leave). >>>> >>>> Cheers, >>>> >>>> Tom >>>> >>>> >>>> >>>> On Sat, 25 Apr 2009 11:50:10 -0000, Peter Firmstone >>>> >>>> >>>> >>> >>> >>>> wrote: >>>> >>>> >>>> >>>> >>>>> I hear you. >>>>> >>>>> How about we step back, postpone making a decision until we better >>>>> understand just what we've inherited then weigh up the pro's and con's? I >>>>> haven't seen anyone in a hurry to write junit tests yet, a little wait while >>>>> we investigate jtreg can't do any harm? ;) It took me about 1 hour to >>>>> download, install and get jtreg running, having never heard of it before, >>>>> the tests themselves seem simple, remember I'm one of your new developers. >>>>> >>>>> I'm going to sit on the fence for now have a play and write test cases >>>>> for ClassDep using both tools for my own understanding. >>>>> >>>>> We're going to have to get to know jtreg to make an informed decision, >>>>> here's some info: >>>>> >>>>> All tests are placed in the directory tree under qa/jtreg/ >>>>> >>>>> Ant can be used to run all tests, I'll post the build.xml file after I >>>>> write it. >>>>> >>>>> The following I've copied from the jtreg website: >>>>> >>>>> Ant Examples: >>>>> >>>>> >>>>> >>>>> Run all the tests in the test/tools/javac directory, using the same JDK >>>>> being used to run Ant, the default work and report directories, and running >>>>> each test in a separate VM. >>>>> >>>>> >>>>> >>>>> As before, but specifying a work directory for the results. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Run just the api/*.java tests. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Run the test/tools/javac tests using /java/jdk/1.5.0 writing the results >>>>> to myWorkDir. >>>>> >>>>> 2. Writing a JDK Regression Test >>>>> 2.1. How do I write a test? >>>>> >>>>> The simplest test is an ordinary Java program with the usual static main >>>>> method. If the test fails, it should throw an exception; if it succeeds, it >>>>> should return normally. >>>>> >>>>> Here's an example: >>>>> >>>>> /* @test 1.1 97/10/12 >>>>> @bug 1234567 >>>>> @summary Make sure that 1 != 0 >>>>> */ >>>>> >>>>> public class OneZero { >>>>> >>>>> public static void main(String[] args) >>>>> throws Exception { >>>>> if (1 == 0) { >>>>> throw new Exception("1 == >>>>> >>>>> >>>>> >>>> 0"); >>>> >>> >>>> } >>>> >>>>> } >>>>> >>>>> } >>>>> The @test tag identifies a source file that >>>>> defines a test. the harness will automatically run any .java, .sh, and .html >>>>> file containing an @test tag within the appropriate comment; it ignores any >>>>> file not containing such a tag or not utilizing one of the expected >>>>> extensions. >>>>> >>>>> If necessary the harness will compile the source file, if the class >>>>> files are older than the corresponding source files. Other files which the >>>>> test depends on must be specified with the @run build >>>>> >>>>> >>>>> >>>> action. >>>> >>> >>>> The arguments to the @test tag are ignored by the harness. For >>>> >>>>> identification it's useful to put the usual SCCS ID keywords (I and E, each >>>>> letter surrounded by %) after the @test tag. >>>>> >>>>> While not part of the tag specification, some tests use the string >>>>> "/nodynamiccopyright" after @test to indicate that that the file should not >>>>> be subject to automated copyright processing that might affect the operation >>>>> of the test, for example, by affecting the line numbers of the test source >>>>> code. >>>>> 2.3. What do the other tags mean? >>>>> >>>>> The other tags shown above are optional. >>>>> >>>>> The @bug tag should be followed by one or more bug numbers, separated >>>>> >>>>> >>>>> >>> >>>> by spaces. The bug number is useful in diagnosing test failures. It's >>>> >>>>> >>> >>>> OK to write tests that don't have bug numbers, but if you're writing a >>>> >>>>> test for a specific bug please include its number in an @bug tag. >>>>> >>>>> The @summary tag describes the condition that is checked by the test. >>>>> >>>>> >>>>> >>> >>>> It is especially useful for non-regression tests, which by definition >>>> >>>>> >>> >>>> don't have bug numbers, but even if there's a bug number it's helpful >>>> >>>>> >>> >>>> to include a summary. Note that a test summary is generally not the same >>>> >>>>> thing as a Bugtraq synopsis, since the latter describes the bug rather than >>>>> the condition that the bug violates. >>>>> >>>>> >>>>> >>>>> Jeff Ramsdale wrote: >>>>> >>>>> >>>>> >>>>>> Not to mention tool support--IDEs, etc... >>>>>> -j >>>>>> >>>>>> On Fri, Apr 24, 2009 at 6:13 AM, Jukka Zitting < >>>>>> jukka.zitting@gmail.com>wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On Fri, Apr 24, 2009 at 10:06 AM, Peter Firmstone >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>> >>>> wrote: >>>> >>>>>>> >>>>>>> >>>>>>>> It appears to me that I should perhaps be writing jtreg style unit >>>>>>>> >>>>>>>> >>>>>>>> >>> >>>> tests >>>> >>>>>>>> >>>>>>>> >>>>>>> for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> ClassDep rather than JUnit? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> I'm not so sure about that. I can't tell which approach is better >>>>>>> technically as I don't know jtreg, but from a community building >>>>>>> perspective it would be much better if we used something that the >>>>>>> >>>>>>> >>>>>>> >>>>>> vast >>>>>> >>> >>>> majority of the Java world is already familiar with. >>>> >>>>>>> BR, >>>>>>> >>>>>>> Jukka Zitting >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>> www.sucden.co.uk >>> Sucden (UK) Limited, 5 London Bridge Street, London SE1 9SG >>> Telephone +44 20 7940 9400 >>> Registered in England no. 1095841 >>> VAT registration no. GB 446 9061 33 >>> Authorised and Regulated by the Financial Services Authority (FSA) and >>> entered in the FSA register under no. 114239 >>> >>> This email, including any files transmitted with it, is confidential and >>> may be privileged. It may be read, copied and used only by the intended >>> recipient. If you are not the intended recipient of this message, please >>> notify postmaster@sucden.co.uk immediately and delete it from your >>> computer system. >>> >>> We believe, but do not warrant, that this email and its attachments are >>> virus-free, but you should check. >>> Sucden (UK) Ltd may monitor traffic data of both business and personal >>> emails. By replying to this email, you consent to Sucden’s monitoring the >>> content of any emails you send to or receive from Sucden. Sucden is not >>> liable for any opinions expressed by the sender where this is a non-business >>> email. >>> The contents of this e-mail do not constitute advice and should not be >>> regarded as a recommendation to buy, sell or otherwise deal with any >>> particular investment. >>> This message has been scanned for viruses by Mimecast. >>> >>> >>> >> > >