hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bruno P. Kinoshita" <brunodepau...@yahoo.com.br>
Subject Re: [Test-Patch TLP] consensus on naming
Date Tue, 14 Jul 2015 10:08:51 GMT
Hi
Has anyone considered using TAP (Test Anything Protocol) for test reporting? [1][2]
disclaimer: I'm the maintainer of the Jenkins TAP plug-in and tap4j Java library

I know that pointing at something created around 1988 to fix a problem with the current testing
tools may sound weird, but I think using TAP rather than JUnit could work. 

# Test reporting

The syntax is human friendly.

```
1..3ok 1 - everything working as expected
not ok 2 - something went wrongok 3```

# Concurrency
Even though TAP states that is must support concurrency, we are discussing how to make it
more explicit in the next spec [3], what doesn't make it impossible to report tests concurrently
in near real time with TAP 13, with limitations to test order, test suites, etc.

# Logging more information in the test output
With JUnit/TestNG/xUnit it can be quite hard to include extra information to the test output.
Using Jenkins, you can include extra information using a not standard syntax and the JUnit
Attachments Plug-in [4]. But with TAP you can use YAMLish [5][6]. It was never included in
any TAP specification, but has been the most widely used way to extend test results with TAP
(I started using it in 2010).
```
1..3ok 1 - everything working as expectednot ok 2 - something went wrong---  date: 2001-11-23
15:03:17 -5
  user: ed
  fatal:
    Unknown variable "bar"
  stack:
    - file: TopClass.py
      line: 23
      code: |
        x = MoreObject("345\n")
    - file: MoreClass.py
      line: 58
      code: |-
        foo = bar...ok 3
Jenkins TAP Plug-in [7] is an example of how to extend Jenkins to support different report
formats. The YAMLish can already be used to attach files in the test result. It uses tap4j
[8], which would have to be changed to support tests in parallel, but it wouldn't be so hard.


It is just an example, so it could be possible to write a custom reporter and not use the
TAP plug-in or tap4j, but since Perl has used TAP for so long, and people have tried running
tests in parallel with Test::More, Test::Parallel and prove, maybe something could be re-used
for Hadoop? I think Git is also using TAP to report its tests, though I've never compiled
and tested it, and I think MySQL or some of its modules/tools uses TAP as well. Another possibility
would be to take a look at the existing issue in Jenkins for real time tests [9] 

Just my 0.02 cents.

Bruno

[1] http://testanything.org/philosophy.html[2] https://github.com/testanything
[3] https://github.com/TestAnything/Specification/issues/15[4] https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin
[5] https://www.npmjs.com/package/yamlish
[6] https://github.com/TestAnything/Specification/issues/11
[7] https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin
[8] http://tap4j.org/[9] https://issues.jenkins-ci.org/browse/JENKINS-19154


      From: Steve Loughran <stevel@hortonworks.com>
 To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org> 
 Sent: Monday, July 13, 2015 9:52 PM
 Subject: Re: [Test-Patch TLP] consensus on naming
   



> On 12 Jul 2015, at 22:32, Sean Busbey <busbey@cloudera.com> wrote:
> 
> sure. what did you have in mind?
> 
> Last time it was discussed we were going to wait to overhaul the repo until
> we have a new repo to move to.
> 
> On Sat, Jul 11, 2015 at 7:40 AM, Steve Loughran <stevel@hortonworks.com>
> wrote:
> 
>> +1,
>> 
>> could you structure the source tree/build so that adding new modules is
>> easy?


nothing particular, except I want to do something better with test running and reporting

http://steveloughran.blogspot.co.uk/2015/05/distributed-system-testing-where-now.html
http://steveloughran.blogspot.co.uk/2015/06/why-is-so-much-of-my-life-wasted.html

I actually think that Yetus may be the wrong place; I want to do some baby steps with

-something to import ant Junit XML and generate some aggregate (avro?) format
-spark libs for analysis in ipython, zepplin of test runs and test runs across time and configurations.

ultimately I'd like to  be able to stream test events from test runners & log events
from (distributed) processes together for a linearized event log which could go into better
analysis & some webapp to view better, with some as-test-run reporting.

I think overall this wouldn't fit into Yetus, something way downstream -but I might like some
of the test & log reporting in there if that helps get the dependency graph right.

Like I said: want. Spare time work, which is why I didn't propose a talk on it at apachecon
EU data.

   

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message