hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: Build/test infrastructure
Date Sun, 27 Feb 2011 00:03:33 GMT
The proposed test automation process hasn't been thought through.   Apache Hudson has been
setup to trigger patch builds, and setup pre-commit test environment.  Unfortunately, the
current setup needs refinement with proper source code setup to make the builds working again.
 Ideally, the test cycle have a commit build which runs simple unit tests, and a secondary
build (every 24 hours) to run more through tests on multiple machine setup.  The test cluster
should be cleansed after every secondary build, and ideally this is done in a sandbox approach.
 However, I don't think bring in puppet environment setup is making the test system reproducible.
 Consequently, it may be better to have the cluster test setup as part of scripts in maven
integration test phase.  This will enable any hadoop developer to setup his own test cluster
without setup puppet master.  I am not fixated on build and packaging only, but express my
opinions on improving build system and making the system easier to reproduce.

Regards,
Eric

On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

This discussion isn't about build of the product nor about packaging
of it. We are discussing patch validation and snapshot build
infrastructure.

On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
> We should be very careful about the approach that we chosen for build/packaging.  The
current state of hadoop is coupled together due to lack of standardized RPC format.  Once
this issue is cleared, the community will want to split hdfs and m/r into separated projects
at some point.  It may be better to ensure project is modularized, and work from the same
svn repository.  Maven is great for doing this, and most of the build and scripts can be defined
in pom.xml.  Deployment/test server configuration can be pass in from hudson.  We should ensure
that build and deployment script do not further couple the project.
>
> Regards,
> Eric
>
> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
>> +1.
>>
>> Once HADOOP-7106 is committed, I'd like to propose we create a directory at the same
level of common/hdfs/mapreduce to hold build (and deploy) type scripts and files.  These would
then get branches/tagged with the rest of the release.
>
> That makes sense, although I don't see changes of the host
> configurations to happen very often.
>
> Cos
>
>> Nige
>>
>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>>
>>> Looking at re-occurring build/test-patch problems on hadoop? build machines I
>>> thought of a way to make them:
>>>  a) all the same (configuration, installed software wise)
>>>  b) have an effortless system to run upgrades/updates on all of them in a
>>>  controlled fashion.
>>>
>>> I would suggest to create Puppet configs (the exact content to be defined)
>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's software
>>> is needed to be restored/updated a simple run of Puppet across the machines
>>> or change in config and run of Puppet will do the magic for us.
>>>
>>> If there are no objections from the community I can put together some
>>> Puppet recipes which might be evolved as we go.
>>>
>>> --
>>> Take care,
>>>       Cos
>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>
>>> After all, it is only the mediocre who are always at their best.
>>>                Jean Giraudoux
>>
>>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message