From dev-return-312445-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Thu Feb 22 07:55:26 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8B1B518064E for ; Thu, 22 Feb 2018 07:55:25 +0100 (CET) Received: (qmail 84305 invoked by uid 500); 22 Feb 2018 06:55:23 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 84290 invoked by uid 99); 22 Feb 2018 06:55:23 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Feb 2018 06:55:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A338DC1C82 for ; Thu, 22 Feb 2018 06:55:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.898 X-Spam-Level: * X-Spam-Status: No, score=1.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id zovWtsUK2I-N for ; Thu, 22 Feb 2018 06:55:19 +0000 (UTC) Received: from mail-yb0-f175.google.com (mail-yb0-f175.google.com [209.85.213.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2AC5E5F3CC for ; Thu, 22 Feb 2018 06:55:18 +0000 (UTC) Received: by mail-yb0-f175.google.com with SMTP id i13-v6so1412822ybl.9 for ; Wed, 21 Feb 2018 22:55:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=HIWoVbPoeXggqWWmDdJEtQ+Vt6VHYigxPdC0//4LR1A=; b=R9uoFdNoK9ORJRsKeCE7v5JzFQLrqa+tqlt3GRgi0vP3dFKgPQmcMhVyxmpKm5/kSN tPbZuB0nZFFXlmgLAskqy5ZY9TDte00ieoNYLXhPLEF8+l//duy7XEruaUTKiKPupMNB zv/3gk9a3n3FpHVNMmoltcG9s/c1i+T3bGph7Zaug6dfXAKcKTuMTMBDlmjC3uA+21bJ mL6ZOaJ3+DOFx3CgyPbyw6e+xJ2+tL4o+ipSrRnZBqP9LkFly+GuXrEds8SZgbZp7enz httZi9Yf6f/A6K5j3ANsOJ0E7QmNBd2MV2dK08Awt7Hdg29eVcALbneq07uu5LGgYULG p4IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=HIWoVbPoeXggqWWmDdJEtQ+Vt6VHYigxPdC0//4LR1A=; b=m3UiGqHBQW31c/TEDRXogQNyLxX4qlLgLs5M41JQUIR8XhTiXbHNKbiPHBYpuU3/gE nzgWA7+nBV+ea/LpslHf6mBRqo2gXi0Aj9gY1u9bdzvKsPhra8JY2eHQF78RyjUtzpU1 BWYJzPKZ9Bxt6qharfvcnOdjYti37JQqvIe1shOunMpfnGXJj8uABV9mNeFmpV0aYnf6 CbJHIfF+KgGYb1qDyC3HHUY4+vDPHk3SEBXJKL2CbkN3HnUvbckzYKsxHKW2x2n4dvFY XTXN9/l3COoyUfLjhTNMATM+SCXV9k1vgf+tgJTO9qzYqNn67B487SYPMCaeRAaYmCPw 1Nxw== X-Gm-Message-State: APf1xPB+t2WtE1np+MGrKRARIzYV+StgJpToZlTZbbs3fkdXzDQVOtP5 pcepxe5HnjXdoFqFpaEjWh5B/WM+mLCIsVhHcQy6hQ== X-Google-Smtp-Source: AH8x22496wGmvPwIpiZ+60i1IL3z85Mo13UAgnv8R14pV5H3mVYDIU/nm2q40l3EGe8QX3NjzMmn5ZHbydg7uB92IxQ= X-Received: by 2002:a25:d347:: with SMTP id e68-v6mr4206927ybf.103.1519282516890; Wed, 21 Feb 2018 22:55:16 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a25:4c5:0:0:0:0:0 with HTTP; Wed, 21 Feb 2018 22:55:15 -0800 (PST) Received: by 2002:a25:4c5:0:0:0:0:0 with HTTP; Wed, 21 Feb 2018 22:55:15 -0800 (PST) In-Reply-To: References: <012701d3ab68$2c3c3c90$84b4b5b0$@thetaphi.de> From: Dawid Weiss Date: Thu, 22 Feb 2018 07:55:15 +0100 Message-ID: Subject: Re: Test failures are out of control...... To: dev@lucene.apache.org Content-Type: multipart/alternative; boundary="000000000000bb991d0565c7868c" --000000000000bb991d0565c7868c Content-Type: text/plain; charset="UTF-8" Don't know, Yonik... If I make a change I am interested in regressions from the start state; running flaky tests makes it impossible and frustrating (and pointless in my opinion). I don't think my expectations are that much off from the average - you may wake up being the only person who has the defaults enabled, which seems wrong. Dawid On Feb 22, 2018 01:55, "Chris Hostetter" wrote: > > : * Hoss has worked on aggregating all test failures from the 3 Jenkins > : systems (ASF, Policeman, and Steve's), downloading the test results & > logs, > : and running some reports/stats on failures. He should be ready to share > : this more publicly soon. > > I think Steve's linked to some of this before from jira comments, but it > was only recently I realized i've never explicitly said to the list "Hey > folks, here's a thing i've been working on" ... > > http://fucit.org/solr-jenkins-reports/ > https://github.com/hossman/jenkins-reports/ > > The most interesting bit is probably here... > > http://fucit.org/solr-jenkins-reports/failure-report.html > > ...but there are currently a few caveats: > > 1) there's some noise inthe '7days' data because I wasn't accounting for > the way jenkins reports some types of failure -- that will gradually clean > itself up > > 2) I think i've been been blocked by builds.apache.org, so at the moment > the data seems to just be from the sarowe & policeman jenkins failures. > > 3) allthough the system is archiving the past 7 days worth of jenkins logs > for any jobs with failures, there is currently no easy way to download > the relevant log(s) from that failure report -- you currently have to > download a CSV file like this one to corrolate the test failures to the > jenkins job, and then go look for that job in the job-data dirs... > > http://fucit.org/solr-jenkins-reports/reports/7days-method-failures.csv > http://fucit.org/solr-jenkins-reports/job-data/ > > (My hope is to make #3 trivial from failure-report.html -- so you can say > "hey weird, this test has failed X times, let's go download those logs." > right from a single screen in your browser) > > > > > -Hoss > http://www.lucidworks.com/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: dev-help@lucene.apache.org > > --000000000000bb991d0565c7868c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Don't k= now, Yonik... If I make a change I am interested in regressions from the st= art state; running flaky tests makes it impossible and frustrating (and poi= ntless in my opinion). I don't think my expectations are that much off = from the average - you may wake up being the only person who has the defaul= ts enabled, which seems wrong.

Dawid


<= /div>

On Feb 22, 2= 018 01:55, "Chris Hostetter" <hossman_lucene@fucit.org> wrote:

: * Hoss has worked on aggregating all test failures from the 3 Jenkins
: systems (ASF, Policeman, and Steve's), downloading the test results &= amp; logs,
: and running some reports/stats on failures. He should be ready to share : this more publicly soon.

I think Steve's linked to some of this before from jira comments, but i= t
was only recently I realized i've never explicitly said to the list &qu= ot;Hey
folks, here's a thing i've been working on" ...

=C2=A0 http://fucit.org/solr-jenkins-reports/
=C2=A0 https://github.com/hossman/jenkins-reports/

The most interesting bit is probably here...

=C2=A0 http://fucit.org/solr-jenkins-r= eports/failure-report.html

...but there are currently a few caveats:

1) there's some noise inthe '7days' data because I wasn't a= ccounting for
the way jenkins reports some types of failure -- that will gradually clean<= br> itself up

2) I think i've been been blocked by builds.apache.org, so at the mo= ment
the data seems to just be from the sarowe & policeman jenkins failures.=

3) allthough the system is archiving the past 7 days worth of jenkins logs<= br> for any jobs with failures, there is currently no easy way to download
the relevant log(s) from that failure report -- you currently have to
download a CSV file like this one to corrolate the test failures to the
jenkins job, and then go look for that job in the job-data dirs...

=C2=A0 http://fucit.org/solr-= jenkins-reports/reports/7days-method-failures.csv
=C2=A0 http://fucit.org/solr-jenkins-reports/job= -data/

(My hope is to make #3 trivial from failure-report.html -- so you can say "hey weird, this test has failed X times, let's go download those = logs."
right from a single screen in your browser)




-Hoss
http://www.lucidworks.com/

-----------------------------------------------------------------= ----
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

--000000000000bb991d0565c7868c--