hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: Build/test infrastructure
Date Sun, 27 Feb 2011 01:38:54 GMT
On 2/26/11 4:34 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

> Apparently you are talking about something else, but I will bite...
> 
> On Sat, Feb 26, 2011 at 04:03PM, Eric Yang wrote:
>> The proposed test automation process hasn't been thought through.   Apache
>> Hudson has been setup to trigger patch builds, and setup pre-commit test
>> environment.  Unfortunately, the current setup needs refinement with proper
>> source code setup to make the builds working again.  Ideally, the test cycle
>> have a commit build which runs simple unit tests, and a secondary build
>> (every 24 hours) to run more through tests on multiple machine setup.  The
>> test cluster should be cleansed after every secondary build, and ideally
> 
> We don't have a test cluster for Apache Hadoop validation. All I am focusing
> on is build and patch validation infrastructure.

If the plan is using puppet agent without puppet master for configuring the
system locally to test patch builds.  It is probably using the wrong tool
for the job.  The value of puppet is to be able to configure heterogeneous
services across machines in a consistent manner.  Is there plan to deploy
multiple services across machines?  If the purpose is using puppet for
config templates, ant or maven can do the job equally well.

> Doing deployment from a build system is certainly possible, but is suboptimal
> because it pollutes the build with HW/OS details, deployment scripts and such.
> Besides, last time I've checked Hadoop was built by Ant.

Deploy to remote machine can be as simple as scp tarball, extra, apply
template, and run it.  None of this requires puppet.  Instead of ant +
puppet combination, the patch test build structure could be simplified by
using maven + shell scripts.

Regards,
Eric

> You don't need to setup puppet muster in order to bounce a node. Puppet works
> i a client-only mode just as perfect.
> 
> Cos
> 

>> packaging only, but express my opinions on improving build system and making
>> the system easier to reproduce.
>> 
>> Regards,
>> Eric
>> 
>> On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>> 
>> This discussion isn't about build of the product nor about packaging
>> of it. We are discussing patch validation and snapshot build
>> infrastructure.
>> 
>> On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
>>> We should be very careful about the approach that we chosen for
>>> build/packaging.  The current state of hadoop is coupled together due to
>>> lack of standardized RPC format.  Once this issue is cleared, the
>>> community will want to split hdfs and m/r into separated projects at some
>>> point.  It may be better to ensure project is modularized, and work from
>>> the same svn repository.  Maven is great for doing this, and most of the
>>> build and scripts can be defined in pom.xml.  Deployment/test server
>>> configuration can be pass in from hudson.  We should ensure that build and
>>> deployment script do not further couple the project.
>>> 
>>> Regards,
>>> Eric
>>> 
>>> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>> 
>>> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
>>>> +1.
>>>> 
>>>> Once HADOOP-7106 is committed, I'd like to propose we create a directory
at
>>>> the same level of common/hdfs/mapreduce to hold build (and deploy) type
>>>> scripts and files.  These would then get branches/tagged with the rest of
>>>> the release.
>>> 
>>> That makes sense, although I don't see changes of the host
>>> configurations to happen very often.
>>> 
>>> Cos
>>> 
>>>> Nige
>>>> 
>>>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>>>> 
>>>>> Looking at re-occurring build/test-patch problems on hadoop? build
>>>>> machines I
>>>>> thought of a way to make them:
>>>>>  a) all the same (configuration, installed software wise)
>>>>>  b) have an effortless system to run upgrades/updates on all of them
in a
>>>>>  controlled fashion.
>>>>> 
>>>>> I would suggest to create Puppet configs (the exact content to be defined)
>>>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's software
>>>>> is needed to be restored/updated a simple run of Puppet across the
>>>>> machines
>>>>> or change in config and run of Puppet will do the magic for us.
>>>>> 
>>>>> If there are no objections from the community I can put together some
>>>>> Puppet recipes which might be evolved as we go.
>>>>> 
>>>>> --
>>>>> Take care,
>>>>>       Cos
>>>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>>> 
>>>>> After all, it is only the mediocre who are always at their best.
>>>>>                Jean Giraudoux
>>>> 
>>>> 
>>> 
>>> 
>> 


Mime
View raw message