hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: Build/test infrastructure
Date Sun, 27 Feb 2011 00:34:15 GMT
Apparently you are talking about something else, but I will bite...

On Sat, Feb 26, 2011 at 04:03PM, Eric Yang wrote:
> The proposed test automation process hasn't been thought through.   Apache
> Hudson has been setup to trigger patch builds, and setup pre-commit test
> environment.  Unfortunately, the current setup needs refinement with proper
> source code setup to make the builds working again.  Ideally, the test cycle
> have a commit build which runs simple unit tests, and a secondary build
> (every 24 hours) to run more through tests on multiple machine setup.  The
> test cluster should be cleansed after every secondary build, and ideally

We don't have a test cluster for Apache Hadoop validation. All I am focusing
on is build and patch validation infrastructure.

> this is done in a sandbox approach.  However, I don't think bring in puppet
> environment setup is making the test system reproducible.  Consequently, it

If a specialized and highly scalable host configuration system such as Puppet
doesn't guarantee configuration reproducibility then I am not sure what else
will. Say, Y! uses that proprietary Igor environment exactly for these
purposes but of course it is highly coupled with yinst format and can't be
used anywhere else.

> may be better to have the cluster test setup as part of scripts in maven
> integration test phase. This will enable any hadoop developer to setup his

Doing deployment from a build system is certainly possible, but is suboptimal
because it pollutes the build with HW/OS details, deployment scripts and such.
Besides, last time I've checked Hadoop was built by Ant.

> own test cluster without setup puppet master. I am not fixated on build and

You don't need to setup puppet muster in order to bounce a node. Puppet works
i a client-only mode just as perfect. 


> packaging only, but express my opinions on improving build system and making
> the system easier to reproduce.
> Regards,
> Eric
> On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
> This discussion isn't about build of the product nor about packaging
> of it. We are discussing patch validation and snapshot build
> infrastructure.
> On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
> > We should be very careful about the approach that we chosen for
> > build/packaging.  The current state of hadoop is coupled together due to
> > lack of standardized RPC format.  Once this issue is cleared, the
> > community will want to split hdfs and m/r into separated projects at some
> > point.  It may be better to ensure project is modularized, and work from
> > the same svn repository.  Maven is great for doing this, and most of the
> > build and scripts can be defined in pom.xml.  Deployment/test server
> > configuration can be pass in from hudson.  We should ensure that build and
> > deployment script do not further couple the project.
> >
> > Regards,
> > Eric
> >
> > On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
> >
> > On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
> >> +1.
> >>
> >> Once HADOOP-7106 is committed, I'd like to propose we create a directory at
the same level of common/hdfs/mapreduce to hold build (and deploy) type scripts and files.
 These would then get branches/tagged with the rest of the release.
> >
> > That makes sense, although I don't see changes of the host
> > configurations to happen very often.
> >
> > Cos
> >
> >> Nige
> >>
> >> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
> >>
> >>> Looking at re-occurring build/test-patch problems on hadoop? build machines
> >>> thought of a way to make them:
> >>>  a) all the same (configuration, installed software wise)
> >>>  b) have an effortless system to run upgrades/updates on all of them in
> >>>  controlled fashion.
> >>>
> >>> I would suggest to create Puppet configs (the exact content to be defined)
> >>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's software
> >>> is needed to be restored/updated a simple run of Puppet across the machines
> >>> or change in config and run of Puppet will do the magic for us.
> >>>
> >>> If there are no objections from the community I can put together some
> >>> Puppet recipes which might be evolved as we go.
> >>>
> >>> --
> >>> Take care,
> >>>       Cos
> >>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >>>
> >>> After all, it is only the mediocre who are always at their best.
> >>>                Jean Giraudoux
> >>
> >>
> >
> >

View raw message