Return-Path: X-Original-To: apmail-hadoop-common-dev-archive@www.apache.org Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52987174B5 for ; Tue, 27 Jan 2015 23:00:26 +0000 (UTC) Received: (qmail 83863 invoked by uid 500); 27 Jan 2015 23:00:25 -0000 Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org Received: (qmail 83783 invoked by uid 500); 27 Jan 2015 23:00:25 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 83338 invoked by uid 99); 27 Jan 2015 23:00:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2015 23:00:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of andrew.wang@cloudera.com designates 209.85.220.177 as permitted sender) Received: from [209.85.220.177] (HELO mail-vc0-f177.google.com) (209.85.220.177) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2015 23:00:09 +0000 Received: by mail-vc0-f177.google.com with SMTP id hy4so5690122vcb.8 for ; Tue, 27 Jan 2015 14:59:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=JOrun9Jvnk0RcZc+4HgBCELA/L5IiMhlDZkomzgVs5w=; b=SR4u3gdjpEYkDXXzKLLAFB3torhZZ+mlUcKKcvDC94b/eJxCdh3WYXEtrFeKTWxs09 H8eGH3Hakohj4FG4/W6gPY8fDoLwNOvLbMjIKMyLECn/TBuoFPUNfeYq5FFxwUn2UKTa 1Df+A83Qh7NuVTFX9jaDxnYao3xCsBcTfEbpi761I5kTTexAhsfpeiCE4VmYNwBd06AX cUyOkg3+03l2F0xA/VW3gpEKfnRUVIJECZWwSDePbENqLZBu/t2HWVEDR2Ou4l6NF5dX dpyqI8DCkyKjUsfJrSu7wyG/hYYz8cpw5S2KypJdZSoOAJ9BoF2avemvlg389t8cI7ud ZcMQ== X-Gm-Message-State: ALoCoQmqQIFNJOQ/57/ie+MJZouS8wE0tPbmuALZFcJ9ooQ7e3WO0Uk2YGmdr+ukk6ePSBvlO2Zu X-Received: by 10.52.31.162 with SMTP id b2mr283387vdi.78.1422399588117; Tue, 27 Jan 2015 14:59:48 -0800 (PST) MIME-Version: 1.0 Received: by 10.52.135.173 with HTTP; Tue, 27 Jan 2015 14:59:27 -0800 (PST) In-Reply-To: References: <2084487980.190020.1422237787978.JavaMail.yahoo@mail.yahoo.com> From: Andrew Wang Date: Tue, 27 Jan 2015 14:59:27 -0800 Message-ID: Subject: Re: Patch review process To: "common-dev@hadoop.apache.org" Content-Type: multipart/alternative; boundary=bcaec51b9a052dc638050daa367e X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51b9a052dc638050daa367e Content-Type: text/plain; charset=UTF-8 > > Andrew, can the community build on your distributed pre-commit work to make > it production ready? > > I'm happy to share it if someone is willing to take it across the finish line. I think it'd be about two weeks of work full-time. I've already cleaned it up some, but it still requires checking out a couple forked repos, and it's strung together with a set of bash scripts. I see the biggest challenges as the following though: - Need to write a faster Swarming client. Running the unit tests just takes 5 mins, but using the Python client to package up all the dependencies takes 15-20 minutes. - It needs to be deployed on Apache infra, integrated with Jenkins JUnit reporting / artifact archiving, and then hooked up to the precommit scripts. I also worry about the ongoing maintenance cost. Brock set up something like this for Hive, and he had a hard time with flaky EC2 instances. One additional upside though is this same framework should work for other mavenized projects too. There's nothing Hadoop specific in it right now, so HBase, Solr, etc could leverage it too. Best, Andrew --bcaec51b9a052dc638050daa367e--