hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: MR1 next steps
Date Tue, 26 Jul 2011 11:22:40 GMT
On 26/07/11 01:23, Arun C Murthy wrote:
> On Jul 25, 2011, at 11:59 AM, Eli Collins wrote:
>> The current plan is to remove the MR1 code from trunk after merging in MR-279.
> One issue is that lots of MR 'unit tests' rely on MiniMRCluster which exposes very minute
aspects of the JT/TT - something we can't support in MR2. Given that we might not have the
time to port all the tests we might want to keep MR1 but delete the scripts in bin/.

Why does the MiniMR cluster go away in MR2?

I quite like it in my own code as I can set it up for a test run, push 
some tests through it and then tear it down. It's got weaknesses

-doesn't shut down very well, tends to hold locks on build/test/metadata 
or something. This doesn't show up in Hadoop JUnit as every Test class 
runs in its own process, but in my functional tests I do bring up the 
cluster in a separate process and push out work to it, and I need to 
leave the cluster live for the entire test run, as per-test-class 
doesn't work.

-does have hardcoded assumptions that there is a "./build/test" subdir 
under which it should create a new dir (not configurable)

These are minor issues. The key thing is that MiniDFS cluster and MiniMR 
cluster let you bring up a short lived MR cluster without any 
installation overhead and run tests against it. Even if the 
hadoop-mapreduce code can move away from it, it's still invaluable 

View raw message