hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Woodhead <adr...@last.fm>
Subject Re: unit testing with hadoop
Date Thu, 22 Nov 2007 19:02:17 GMT
We have done something like this where we want our unit tests to run 
against a 1-machine "cluster". As a starting point we took 
HadoopTestCase and have written our own modified version of this which 
we configure and tell it whether to run in "local fs" or "mini 
mapreduce" mode. The former is slightly faster but doesn't catch all 
bugs so we have a continuous integration machine setup to perform tests 
in both modes. We have also made some changes so that it only starts and 
stops the cluster when the class is loaded instead of between tests 
which also improves the speed.

Anyway, I would say that having a look at the source code for 
HadoopTestCase is probably your best starting point and hopefully you 
can take it from there!



Eugeny N Dzhurinsky wrote:
> Hello there, we would like to make some tests with hadoop.
> For the tests we would like to have a hadoop filesystem up and configured, so
> using stubs and some mocks of core interfaces we can test the overall storage
> functionality we're about to develop (which would be a part of map/reduce jobs
> later).
> As far as we learned from documentation/tutorials, to run hadoop we need to
> use it's own startup scripts, which looks like an overhead and a bit
> cumbersome to integrate in maven testing.
> Could you please advice?

View raw message