Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 85F76200C5B for ; Thu, 13 Apr 2017 00:57:28 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 84615160BA8; Wed, 12 Apr 2017 22:57:28 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CB9FD160B95 for ; Thu, 13 Apr 2017 00:57:27 +0200 (CEST) Received: (qmail 80013 invoked by uid 500); 12 Apr 2017 22:57:26 -0000 Mailing-List: contact dev-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zookeeper.apache.org Delivered-To: mailing list dev@zookeeper.apache.org Received: (qmail 80002 invoked by uid 99); 12 Apr 2017 22:57:26 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Apr 2017 22:57:26 +0000 Received: from mail-lf0-f53.google.com (mail-lf0-f53.google.com [209.85.215.53]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 3A1121A036B for ; Wed, 12 Apr 2017 22:57:26 +0000 (UTC) Received: by mail-lf0-f53.google.com with SMTP id h125so21576168lfe.0 for ; Wed, 12 Apr 2017 15:57:26 -0700 (PDT) X-Gm-Message-State: AN3rC/5lORd4BS3i8n8P2bKTAfnMZk1/NC9ZAa4C7ipMDcDF+1nRirYh rDSZJN7RoLdnMKpeMLYmSrTtNro0BQ== X-Received: by 10.25.56.65 with SMTP id d1mr33185lfj.30.1492037844731; Wed, 12 Apr 2017 15:57:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.225.13 with HTTP; Wed, 12 Apr 2017 15:56:44 -0700 (PDT) In-Reply-To: References: From: Patrick Hunt Date: Wed, 12 Apr 2017 15:56:44 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Introduce ZooKeeper flaky test dashboard To: DevZooKeeper Content-Type: multipart/alternative; boundary=f403045ea68aba0eb7054d002110 archived-at: Wed, 12 Apr 2017 22:57:28 -0000 --f403045ea68aba0eb7054d002110 Content-Type: text/plain; charset=UTF-8 Great stuff, thanks Michael. Is there a way to indicate, for each of the jobs, how many times the job failed vs the number of times it was run during the time period? I often see that when one test fails it will cause others to fail... Patrick On Tue, Apr 11, 2017 at 8:07 PM, Edward Ribeiro wrote: > Very cool feature! Congratulations for initiative! :D > > Em 11 de abr de 2017 11:48 PM, "Jordan Zimmerman" < > jordan@jordanzimmerman.com> escreveu: > > > Nice work - we could really use this on Curator. I'll be stealing this > > soon ;D > > > > > On Apr 11, 2017, at 9:33 PM, Michael Han wrote: > > > > > > tl;dr > > > https://builds.apache.org/job/ZooKeeper-Find-Flaky-Tests/ > > lastSuccessfulBuild/artifact/report.html > > > > > > > > > We all know quality is important to this project. One of major > advantage > > of > > > ZooKeeper comparing to other similar solutions is its quality: solid, > > > stable, and well tested. Unit tests are playing an important role for > > ZK's > > > quality so we need take these tests seriously. In particular for failed > > > tests, it's easy to blame they are failed because of flaky but usually > > > there are bugs (sometimes very subtle ones) behind the failures. > > > > > > Inspired by some good work done on HBase project, I've borrowed and > > tweaked > > > their script so it applies to ZooKeeper project. The dashboard is now > up > > > and a Jenkins job is running on a daily base to monitor our builds and > > pick > > > up flaky tests. I hope this dashboard can help increase the visibility > > and > > > raise awareness of the quality of unit tests, and also help developers > > > track and diagnose failures. > > > > > > Any feedback will be appreciated, and of course, contributions are very > > > welcome. > > > > > > -- > > > Cheers > > > Michael. > > > > > --f403045ea68aba0eb7054d002110--