hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Re: Developing, Testing, Distributing
Date Thu, 07 Apr 2011 14:55:56 GMT
On 04/07/2011 03:39 AM, Guy Doulberg wrote:
> Hey,
> I have been developing Map/Red jars for a while now, and I am still not comfortable with
the developing environment I gathered for myself (and the team)
> I am curious how other Hadoop developers out-there, are developing their jobs...
> What IDE you are using,


> What plugins to the IDE you are using

Um .... subclipse.  (And findbugs sometimes.)

> How do you test your code, which Unit test libraries your using, how do you run your
automatic tests after you have finished the development?

JUnit.  Run the tests right inside eclipse using the IDE's built-in 
junit capabilities.

> Do you have test/qa/staging environments beside the dev and the production? How do you
keep it similar to the production

We have small dev and qa Hadoop clusters, in addition to the large 
production cluster.  We don't do anything particular to keep them 
similar.  If you want to run a test job, and require some data that's on 
the prod cluster, you have to port it yourself.

> Code reuse - how do you build components that can be used in other jobs, do you build
generic map or reduce class?

If you do Test Driven Development when you write your code, you wind up 
with components that you can test independently, and then plug into your 
M/R classes.

> I can tell you that I have no answer to the questions above,
> I hope that this post is not too general, but I think the discussion here could be helpful
for newbie and experienced developers all together
> Thanks Guy

View raw message