hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-5621) MapReducer to run junit tests under Hadoop
Date Fri, 03 Apr 2009 11:24:12 GMT
MapReducer to run junit tests under Hadoop

                 Key: HADOOP-5621
                 URL: https://issues.apache.org/jira/browse/HADOOP-5621
             Project: Hadoop Core
          Issue Type: New Feature
    Affects Versions: 0.21.0
            Reporter: Steve Loughran

This is something I mentioned to some people last week, thought I would start a discussion
on it.

We could run junit tests as a MapReduce job with
# a mapper that takes a list of classes, one per line
# extracts the test suite from each class, and then invokes each test method. This would be
a new junit test runner.
# saves the result (and any exceptions) as the output. Also saves any machine specific details.

# It also needs to grab the System.out and System.err channels, to map them to specific tests.
# Measure how long the tests took (incuding setup/teardown time)
# Add an ant task <listresources> to take filesets and other patterns, and generate
text files from the contents (with stripping of prefixes and suffices, directory separator
substition, file begin/end values, etc, etc). I have this with tests already.

The result would be that you could point listresources at a directory tree and create a text
file listing all tests to run. These could be executed across multiple hosts and the results
correlated. It would be, initially, a MapExpand, as the output would be bigger than the input

Feature creep then becomes the analysis

# Add another MR class which runs through all failing tests and creates a new list of test
classes that failed. This could be rescheduled on different runs, and makes for a faster cycle
(only run failing tests until they work)
# Add something to only get failing tests, summarise them (somehow) in a user readable form
# Something to get partially failing tests and highlight machine differences. 
# Add something to compare tests over time, detect those which are getting slower?
# an MR to regenerate the classic Ant junit XML reports, for presentation in other tools (like

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message