hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nigel Daley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using jira_cli tool
Date Thu, 21 Oct 2010 06:42:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923345#action_12923345
] 

Nigel Daley commented on HADOOP-7003:
-------------------------------------

The current patch testing process has a number of flaws:
# reliance on email to determine when a patch is available
# a job per *slave* that you want to run the testing on
# because of 2, poor sharing of resources across many projects
# an admin job that tracks the queue and running jobs but can get confused if jobs fail or
hudson is rebooted

Hudson has come a long way since I first set it up 4 years ago to run our patch testing. 
A recent Hudson feature allows the *same* job to be executed at the *same* time.  A more established
feature is the ability to pass parameters between jobs.  These feature now enable a simpler
architecture for patch testing.

Here are the new Hudson jobs needed:

*PreCommit-Admin*
This job replaces the current patch testing admin job and runs every 5 to 10 minutes.  It
runs a new version of the hudsonPatchQueueAdmin.sh script (attached).  The new script roughly
follows these steps:
# grabs the XML output of a Jira filter that contains all the issues that should be tested
by the system.  For instance, this filter contains all the HADOOP, HDFS, and MAPREDUCE issues
that are in Patch Available state: https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-xml/12313474/SearchRequest-12313474.xml?tempMax=100
 By changing the filter (eg. adding PIG and ZOOKEEPER for instance), this hudson job will
run tests for those project issues as well.  
# processes the XML into a list of <project-issue>, <attachment id> pairs, where
attachment id is the id for the newest attachment
# grabs the one build artifact that this job keeps: a list of pairs: <project-issue>,
<attachment id>.  This file is called patch_tested.txt and contains the pairs already
submitted for testing.
# for each pair from the XML, see if the pair is in patch_tested.txt.  If not, kick a new
project build (next job below) and append the pair to the file.  When kicking off the new
build, pass the issue number (and optional the attachment id) to the build.  The build to
kick off is computed as "PreCommit-<PROJECT NAME>-Build".  This same project build can
be kicked off many times by this execution of the admin job -- each instance sitting in the
Hudson queue until a slave is available.
# when this admin job completes, archive the latest patch_tested.txt

*PreCommit-<PROJECT>-Build*
A job with this name (eg PreCommit-MAPREDUCE-Build) must exist for each Jira project administered
by PreCommit-Admin.  The job must enable concurrent builds in it's configuration.  This job
downloads the patch, does the actual testing and puts a comment in Jira.  It does not communicate
back to the admin job.  The job is tied to slaves that are configured with a certain label.
 The label is the same as the project name.

*Slaves*
Slaves are configured to run a certain project job (like PreCommit-HDFS-Build) by being labeled
with the project (eg HDFS).  This means that any properly configured slave can be easily added
to the build pool and start taking on load.  It also means that most slaves should be able
to run most patch tests for most projects.


> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>         Attachments: hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message