hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kate rhodes" <masuk...@gmail.com>
Subject Re: Unit test Examples for Mapper and Reducer?
Date Fri, 21 Sep 2007 11:52:39 GMT
On 9/21/07, Enis Soztutar <enis.soz.nutch@gmail.com> wrote:
> Hi,
> You can develop your unit tests by using classes under src/test. Infact
> you may find MiniMRCluster and MiniMRDFS very useful. Alternatively you
> may invoke the program in local mode, with a smaller input to check the
> outcomes.

Anything that involves communicating with another class is NOT a unit
test. It's an integration test because problems in that other class
could give false failures. Also, bugs you're unknowingly relying on in
the other class could provide false passes.  If, for example, someone
introduced a bug in MiniMRCluster and I was relying on it in my test
for my Mapper then in would cause my test to fail. And that's bull. It
means I wasn't testing my Mapper, I was testing the integration of my
Mapper with the MiniMRCluster with probably 40 or 50 other classes
that get tied in.

A unit test *has* to be totally isolated. The class under test should
be allowed no communication with any other object that isn't a
mock-object otherwise the problems noted above crop up. I need to know
that the class I'm testing is written correctly regardless of what
other bugs may or may not have been introduced in other classes.
Otherwise I see my class fail and start wondering what I've screwed up
in it, and what needs debugging even if my class is 100% correct. I'm
not willing to waste my time hunting the causes of bogus failures when
i could have a test suite that actually told me the correct class that
had the bug in it.

There are good reasons to write integration tests but you have to
understand that they are dramatically different beasts than unit

I had already found those classes you mentioned, but, like I said, I
haven't been able to find any examples of a UNIT test of a Mapper or
Reducer (integration tests don't count).

- kate = masukomi

View raw message