hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kate rhodes" <masuk...@gmail.com>
Subject Why no mock objects in Hadoop? [was Unit test Examples for Mapper and Reducer?]
Date Sat, 22 Sep 2007 21:59:08 GMT
On 9/21/07, Nigel Daley <ndaley@yahoo-inc.com> wrote:
> All the junit tests in hadoop are system type tests (as you've
> noted).  But, since Mapper and Reducer method parameters are all
> interfaces, it should be easy to use something like EasyMock or jMock
> to mock the input parameters and test your classes.

You are correct... in part. There are two problems that I discovered with this.
1) JobConf does not implement an interface it extends Configuration
and the only thing Configuration implements is Iterable.
2) and this is the scary one... There are NO mock object libraries OR
objects in Hadoop. OMFGWTFBBQ. Seriously guys. This means there's
*probably* not a single UNIT test in Hadoop, just lots of integration
and system tests. Not to diss the integration tests you guys have
written but they're NOT unit tests and that fact means that you're
just hurting yourselves when it comes time to debug because a bug in
one class has the potential to show up as a failure in any of the
other classes that use it in their tests, which can mislead you into
thinking that you've got a bug in a class that's perfectly fine.

Now, I've got no problems using EasyMock and / or jMock for testing my
personal apps I do it all the time, but it means that I can't add any
to Hadoop without convincing someone to add in one of those two jars
(actually in jMock's case it looks like a whole pile-o-jars).

If someone will add EasyMock or jMock to Hadoop I'd be willing to help
refactor some of the existing tests that were intended to be unit
tests. Or, if there's a potential licencing issue with distributing it
with Hadoop at least declare one of them to be a defacto part of
Hadoop (like the missing ant.jar) so that people can start writing
tests that use them.

Speaking of which WHY is the ant.jar not included with Hadoop? It
depends on it, and there's obviously not a licensing issue, but you
can't do half the things in it without copying it into the lib dir.

- kate = masukomi

View raw message