From dev-return-7484-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Tue Mar 31 16:37:32 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A44C2180181 for ; Tue, 31 Mar 2020 18:37:31 +0200 (CEST) Received: (qmail 28864 invoked by uid 500); 31 Mar 2020 16:37:30 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 28851 invoked by uid 99); 31 Mar 2020 16:37:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Mar 2020 16:37:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id ED8D9C1C04 for ; Tue, 31 Mar 2020 16:37:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1 X-Spam-Level: * X-Spam-Status: No, score=1 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id OxqPJWjt2JJM for ; Tue, 31 Mar 2020 16:37:27 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.166.44; helo=mail-io1-f44.google.com; envelope-from=joseph.evans@gmail.com; receiver= Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 3EFEFBB818 for ; Tue, 31 Mar 2020 16:37:27 +0000 (UTC) Received: by mail-io1-f44.google.com with SMTP id i3so13249044ioo.13 for ; Tue, 31 Mar 2020 09:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=tulSuOvhWvOdkb0HMqIBU6rxxKT0HRfTLjLOiajfWik=; b=XHMCVNgs+rBmG+zkGrjDoW2s3zkjQ06QW5qVz74mRFySF3E5VeTiXOEU34QNlY2N4P hhLnuOxZvNsnZrtG9m9XD3nI7QD+T6DWTmcSaLb4O8KZS+bq1ToQfOaMnTSgl4PkZz+A Djorm3vyvoKtSI4XFUqqX6YjvrsZJ0MT/fA6KGTPSHHquXYC6tbdfuEEo6pdJdnFPoGF yWShRlW80Kb9v1esG6GpaeRzUXZ8q7dYeFyv/1XXhO6UDGxmxkL91M7HCdPcCupVw+PV h+2lB6LMR6Eu5EUEi7TzSz0Q2KxnkBhYcy0VxegcO1LLMFXvDf/+F5oklu0grVSZzrRb /6rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=tulSuOvhWvOdkb0HMqIBU6rxxKT0HRfTLjLOiajfWik=; b=Qh1H9qyVdoerXpg05TtAXg7JHIjGimfiDUxLiCS0/uW4+ZoFq5R8NPiHnLpZBSVpvA l4zeG1T7fIUkQ76XETzcHiLh7egRSbZupcnhAQRRKgVHaRNoXRRSxCNBJZ6DrgZpbpJx CJnU9unlFC2WWnbcDuVpOWf1vmJlZ2AJMunMrEnRJjRAfBgQqX86HsMA5nSSK1Iq3uWI pN2Qh3edXgs/nLWEkzY9EX9n6p6Q7/4lVpABuO1USj/ujJgdGCj47q2fqWHgB63H96zc F4fjH9LYw4UH6vPOrcbDu6yjAmAbEd6ORZLOY0Sbvo5Ap3kCOnODXygTeKigVe7pB62G vtjg== X-Gm-Message-State: ANhLgQ19JIDlHGDJgsrsjSD381KUvV2Zbv2EkST0+u0aBw92qOlTvHxV w2MaLhGdU2j0/V32pCWI+bbwKOPjuE4lWlQ2AvrqW2xRtNQ= X-Google-Smtp-Source: ADFU+vsRz0vaNwkFqCUmErzt7KZS9d96KuKBTFTJrccNVOdkvE97vM+cwHvYiRJZ0MBTMg8wEBUrFjl3iqA092wTdeY= X-Received: by 2002:a05:6638:218:: with SMTP id e24mr6789074jaq.48.1585672640281; Tue, 31 Mar 2020 09:37:20 -0700 (PDT) MIME-Version: 1.0 References: <8159bdefcb117056aeaf46096804b7fe71fbec3a.camel@amazon.com> In-Reply-To: From: Joe Evans Date: Tue, 31 Mar 2020 09:37:09 -0700 Message-ID: Subject: Re: CI Pipeline Change Proposal To: dev@mxnet.incubator.apache.org Content-Type: multipart/alternative; boundary="000000000000745a0405a2292ecb" --000000000000745a0405a2292ecb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks everyone for your input. I've created an issue for tracking the increased sanity build time, but this should be treated as a separate project. https://github.com/apache/incubator-mxnet/issues/17945 In the meantime, to keep momentum going on the staggered build pipeline project, please let me know if there are any concerns moving forward. Thanks! On Fri, Mar 27, 2020 at 7:57 AM Marco de Abreu wrote: > The docker cache images can be used by you. They're available in Dockerhu= b, > you just have to tweak the docker run method. > > The thing is that the scripts CI uses has the intention that layers chang= e > and thus the cache is used. > > If you want to be able to change the layers, then you have to accept the > fact that docker build is required. If you are fine with accepting whatev= er > is published in Dockerhub, then you can use docker run. > > The thing which is missing here is a flag in build.py which states whethe= r > either the local dir or the remote cache should be used to create the > docker environment. Your dir would still be mounted into the container, b= ut > the difference is that the layers would not necessarily match to what's i= n > your dir. Speak, if you change the layers scripts, they'd obviously not b= e > available in the image if you consume Dockerhub. > > Would such a feature help you reduce the Painpoints you are encountering? > > With regards to the pinning: I think it is not feasible to not do pinning > but at the same time expect everything to stay the same. I can recommend > tackling that concern by introducing pinning and having an automated syst= em > (or an assigned person) which tests later versions on a repeating base. > > -Marco > > Aaron Markham schrieb am Fr., 27. M=C3=A4rz 2= 020, > 15:48: > > > Sure. That's the fix for now. > > > > But, I've noticed that when that's done and there's no process to enfor= ce > > upgrades and patching, these get really out of date and the problems > > compound. > > > > Plus, when I build locally using docker, I can never seem to get the > > benefit of the cache. Or at least not in the way I'd expect it to be. > > > > On one hand I get to discover all these bugs before they hit prod. Ha. > > But I'd like to have dockerfiles that pull base images that have v1.6.0 > and > > each patched major release. Ideally I'd have ones for each language > binding > > too. > > Seems like that might really simplify trying to work with a particular > > issue. Instead, I'm constantly rebuilding binaries and having to reset > the > > submodules and make clean. > > Seems even more relevant now that we're maintaining master and 1.7.x an= d > at > > least for me, 1.6.x. > > > > > > > > On Fri, Mar 27, 2020, 00:55 Marco de Abreu > > wrote: > > > > > What about dependency pinning? > > > > > > The cache should not be our method to do dependency pinning and > > > synchronization. > > > > > > -Marco > > > > > > Aaron Markham schrieb am Fr., 27. M=C3=A4= rz > 2020, > > > 03:45: > > > > > > > I'm dealing with a Ruby dep breaking the site build right now. > > > > I wish this would be on occasion that I choose, not when Ruby or x > > > > dependency releases a new version. When the cache expires for Jekyl= l > > the > > > > site won't publish anymore... And CI will be blocked for the websit= e > > > test. > > > > > > > > If we built the base OS and main deps once when we do a minor > release, > > > > upload that to dockerhub, then we'd save build time and things > breaking > > > > randomly. Users can use those docker images too. At release time we > do > > a > > > > round of updates and testing when we're ready. Can we find a balanc= e > > > > between caching, prebuilt docker images, freshness, and efficiency? > > > > > > > > > > > > On Thu, Mar 26, 2020, 14:31 Marco de Abreu > > > > wrote: > > > > > > > > > Correct. But I'm surprised about 2:50min to pull down the images. > > > > > > > > > > Maybe it makes sense to use ECR as mirror? > > > > > > > > > > -Marco > > > > > > > > > > Joe Evans schrieb am Do., 26. M=C3=A4rz = 2020, > > > 22:02: > > > > > > > > > > > +1 on rebuilding the containers regularly without caching layer= s. > > > > > > > > > > > > We are both pulling down a bunch of docker layers (when docker > > pulls > > > an > > > > > > image) and then building a new container to run the sanity buil= d > > in. > > > > > > Pulling down all the layers is what is taking so long (2m50s.) > > Within > > > > the > > > > > > docker build, all the layers are cached, so it doesn't take lon= g. > > > > Unless > > > > > > I'm missing something, it doesn't make much sense to be > rebuilding > > > the > > > > > > image every build. > > > > > > > > > > > > On Thu, Mar 26, 2020 at 1:12 PM Lausen, Leonard > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > WRT Docker Cache: We need to add a mechanism to invalidate th= e > > > cache > > > > > and > > > > > > > rebuild > > > > > > > the containers on a set schedule. The builds break too often > and > > > the > > > > > > > breakage is > > > > > > > only detected when a contributor touches the Dockerfiles > > (manually > > > > > > causing > > > > > > > cache > > > > > > > invalidation) > > > > > > > > > > > > > > On Thu, 2020-03-26 at 16:06 -0400, Aaron Markham wrote: > > > > > > > > I think it is a good idea to do the sanity check first. Eve= n > at > > > 10 > > > > > > > minutes. > > > > > > > > And also try to fix the docker cache situation, but those c= an > > be > > > > > > separate > > > > > > > > tasks. > > > > > > > > > > > > > > > > On Thu, Mar 26, 2020, 12:52 Marco de Abreu < > > > > marco.g.abreu@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Jenkins doesn't load for me, so let me ask this way: are = we > > > > > actually > > > > > > > > > rebuilding every single time or do you mean the docker > cache? > > > > > Pulling > > > > > > > the > > > > > > > > > cache should only take a few seconds from my experience - > > > docker > > > > > > build > > > > > > > > > should be a no-op in most cases. > > > > > > > > > > > > > > > > > > -Marco > > > > > > > > > > > > > > > > > > > > > > > > > > > Joe Evans schrieb am Do., 26. > M=C3=A4rz > > > > 2020, > > > > > > > 20:46: > > > > > > > > > > > > > > > > > > > The sanity-lint check pulls a docker image cache, build= s > a > > > new > > > > > > > container > > > > > > > > > > and runs inside. The docker setup is taking around 3 > > minutes, > > > > at > > > > > > > least: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-va= lidation%2Fsanity/detail/master/1764/pipeline/39 > > > > > > > > > > We could improve this by not having to build a new > > container > > > > > every > > > > > > > time. > > > > > > > > > > Also, our CI containers are huge so it takes awhile to > pull > > > > them > > > > > > > down. > > > > > > > > > I'm > > > > > > > > > > sure we could reduce the size by being a bit more caref= ul > > in > > > > > > building > > > > > > > > > them > > > > > > > > > > too. > > > > > > > > > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > On Thu, Mar 26, 2020 at 12:33 PM Marco de Abreu < > > > > > > > marco.g.abreu@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Do you know what's driving the duration for sanity? I= t > > used > > > > to > > > > > be > > > > > > > 50 > > > > > > > > > sec > > > > > > > > > > > execution and 60 sec preparation. > > > > > > > > > > > > > > > > > > > > > > -Marco > > > > > > > > > > > > > > > > > > > > > > Joe Evans schrieb am Do., 26= . > > > M=C3=A4rz > > > > > > 2020, > > > > > > > > > 20:31: > > > > > > > > > > > > Thanks Marco and Aaron for your input. > > > > > > > > > > > > > > > > > > > > > > > > > Can you show by how much the duration will > increase? > > > > > > > > > > > > > > > > > > > > > > > > The average sanity build time is around 10min, whil= e > > the > > > > > > average > > > > > > > > > build > > > > > > > > > > > time > > > > > > > > > > > > for unix-cpu is about 2 hours, so the entire build > > > pipeline > > > > > > would > > > > > > > > > > > increase > > > > > > > > > > > > by 2 hours if we required both unix-cpu and sanity = to > > > > > complete > > > > > > in > > > > > > > > > > > parallel. > > > > > > > > > > > > I took a look at the CloudWatch metrics we're savin= g > > for > > > > > > Jenkins > > > > > > > > > jobs. > > > > > > > > > > > Here > > > > > > > > > > > > is the failure rate per job, based on builds > triggered > > by > > > > PRs > > > > > > in > > > > > > > the > > > > > > > > > > past > > > > > > > > > > > > year. As you can see, the sanity build failure is > still > > > > > fairly > > > > > > > high > > > > > > > > > and > > > > > > > > > > > > would save a lot of unneeded build jobs. > > > > > > > > > > > > > > > > > > > > > > > > Job Successful Failed Failure Rate > > > > > > > > > > > > sanity 6900 2729 28.34% > > > > > > > > > > > > unix-cpu 4268 4786 52.86% > > > > > > > > > > > > unix-gpu 3686 5637 60.46% > > > > > > > > > > > > centos-cpu 6777 2809 29.30% > > > > > > > > > > > > centos-gpu 6318 3350 34.65% > > > > > > > > > > > > clang 7879 1588 16.77% > > > > > > > > > > > > edge 7654 1933 20.16% > > > > > > > > > > > > miscellaneous 8090 1510 15.73% > > > > > > > > > > > > website 7226 2179 23.17% > > > > > > > > > > > > windows-cpu 6084 3621 37.31% > > > > > > > > > > > > windows-gpu 5191 4721 47.63% > > > > > > > > > > > > > > > > > > > > > > > > We can start by requiring only the sanity job to > > complete > > > > > > before > > > > > > > > > > > triggering > > > > > > > > > > > > the rest, and collect data to decide if it makes > sense > > to > > > > > > change > > > > > > > it > > > > > > > > > > from > > > > > > > > > > > > there. Any objections to this approach? > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 25, 2020 at 9:35 AM Marco de Abreu < > > > > > > > > > > marco.g.abreu@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Back then I have created a system which exports a= ll > > > > Jenkins > > > > > > > results > > > > > > > > > > to > > > > > > > > > > > > > cloud watch. It does not include individual test > > > results > > > > > but > > > > > > > rather > > > > > > > > > > > > stages > > > > > > > > > > > > > and jobs. The data for the sanity check should be > > > > available > > > > > > > there. > > > > > > > > > > > > > > > > > > > > > > > > > > Something I'd also be curious about is the > percentage > > > of > > > > > the > > > > > > > > > failures > > > > > > > > > > > in > > > > > > > > > > > > > one run. Speak, if a commit failed, have there be= en > > > > > multiple > > > > > > > jobs > > > > > > > > > > > failing > > > > > > > > > > > > > (indicating an error in the code) or only one or > two > > > > > > > (indicating > > > > > > > > > > > > > flakyness). This should give us a proper > > understanding > > > of > > > > > how > > > > > > > > > > > unnecessary > > > > > > > > > > > > > these runs really are. > > > > > > > > > > > > > > > > > > > > > > > > > > -Marck > > > > > > > > > > > > > > > > > > > > > > > > > > Aaron Markham schrieb > am > > > > Mi., > > > > > > 25. > > > > > > > M=C3=A4rz > > > > > > > > > > > 2020, > > > > > > > > > > > > > 16:53: > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 for sanity check - that's fast. > > > > > > > > > > > > > > -1 for unix-cpu - that's slow and can just hang= . > > > > > > > > > > > > > > > > > > > > > > > > > > > > So my suggestion would be to see the data apart= - > > > > what's > > > > > > the > > > > > > > > > > failure > > > > > > > > > > > > > > rate on the sanity check and the unix-cpu? > > Actually, > > > > can > > > > > we > > > > > > > get a > > > > > > > > > > > > > > table of all of the tests with this data?! > > > > > > > > > > > > > > If the sanity check fails... let's say 20% of t= he > > > time, > > > > > but > > > > > > > only > > > > > > > > > > > takes > > > > > > > > > > > > > > a couple of minutes, then ya, let's stack it an= d > do > > > > that > > > > > > one > > > > > > > > > first. > > > > > > > > > > > > > > I think unix-cpu needs to be broken apart. It's > too > > > > > complex > > > > > > > and > > > > > > > > > > fails > > > > > > > > > > > > > > in multiple ways. Isolate the brittle parts. Th= en > > we > > > > can > > > > > > > > > > > > > > restart/disable those as needed, while all of t= he > > > other > > > > > > parts > > > > > > > > > pass > > > > > > > > > > > and > > > > > > > > > > > > > > don't have to be rerun. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 25, 2020 at 1:32 AM Marco de Abreu = < > > > > > > > > > > > > marco.g.abreu@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > We had this structure in the past and the > > community > > > > was > > > > > > > > > bothered > > > > > > > > > > by > > > > > > > > > > > > CI > > > > > > > > > > > > > > > taking more time, thus we moved to the curren= t > > > model > > > > > with > > > > > > > > > > > everything > > > > > > > > > > > > > > > parallelized. We'd basically revert that then= . > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you show by how much the duration will > > > increase? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also, we have zero test parallelisation, spea= k > we > > > are > > > > > > > running > > > > > > > > > one > > > > > > > > > > > > test > > > > > > > > > > > > > on > > > > > > > > > > > > > > > 72 core machines (although multiple workers). > > > > Wouldn't > > > > > it > > > > > > > be > > > > > > > > > way > > > > > > > > > > > more > > > > > > > > > > > > > > > efficient to add parallelisation and thus > heavily > > > > > reduce > > > > > > > the > > > > > > > > > time > > > > > > > > > > > > spent > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > the tasks instead of staggering? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I feel concerned that these measures to save > cost > > > are > > > > > > paid > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > form > > > > > > > > > > > > > > of a > > > > > > > > > > > > > > > worse user experience. I see a big potential = to > > > save > > > > > > costs > > > > > > > by > > > > > > > > > > > > > increasing > > > > > > > > > > > > > > > efficiency while actually improving the user > > > > experience > > > > > > > due to > > > > > > > > > CI > > > > > > > > > > > > being > > > > > > > > > > > > > > > faster. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Marco > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Joe Evans schrieb am > > Mi., > > > > 25. > > > > > > > M=C3=A4rz > > > > > > > > > > 2020, > > > > > > > > > > > > > 04:58: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > First, I just wanted to introduce myself to > the > > > > MXNet > > > > > > > > > > community. > > > > > > > > > > > > I=E2=80=99m > > > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > and will be working with Chai and the AWS > team > > to > > > > > > improve > > > > > > > > > some > > > > > > > > > > > > issues > > > > > > > > > > > > > > > > around MXNet CI. One of our goals is to > reduce > > > the > > > > > > costs > > > > > > > > > > > associated > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > running MXNet CI. The task I=E2=80=99m work= ing on now > > is > > > > this > > > > > > > issue: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/17802 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Proposal: Staggered Jenkins CI pipeline > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Based on data collected from Jenkins, aroun= d > > 55% > > > of > > > > > the > > > > > > > time > > > > > > > > > > when > > > > > > > > > > > > the > > > > > > > > > > > > > > > > mxnet-validation CI build is triggered by a > PR, > > > > > either > > > > > > > the > > > > > > > > > > sanity > > > > > > > > > > > > or > > > > > > > > > > > > > > > > unix-cpu builds fail. When either of these > > builds > > > > > fail, > > > > > > > it > > > > > > > > > > > doesn=E2=80=99t > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > sense to run the rest of the pipelines and > > > utilize > > > > > all > > > > > > > those > > > > > > > > > > > > > resources > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > we=E2=80=99ve already identified a build or= unit test > > > > > failure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We are proposing changing the MXNet Jenkins > CI > > > > > pipeline > > > > > > > by > > > > > > > > > > > > requiring > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > *sanity* and *unix-cpu* builds to complete > and > > > pass > > > > > > tests > > > > > > > > > > > > > successfully > > > > > > > > > > > > > > > > before starting the other build pipelines > > > > > > > (centos-cpu/gpu, > > > > > > > > > > > > unix-gpu, > > > > > > > > > > > > > > > > windows-cpu/gpu, etc.) Once the sanity buil= ds > > > > > > > successfully > > > > > > > > > > > > complete, > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > remaining build pipelines will be triggered > and > > > run > > > > > in > > > > > > > > > parallel > > > > > > > > > > > (as > > > > > > > > > > > > > > they > > > > > > > > > > > > > > > > currently do.) The purpose of this change i= s > to > > > > > > identify > > > > > > > > > faulty > > > > > > > > > > > > code > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > compatibility issues early and prevent > further > > > > > > execution > > > > > > > of > > > > > > > > > CI > > > > > > > > > > > > > builds. > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > will increase the time required to test a P= R, > > but > > > > > will > > > > > > > > > prevent > > > > > > > > > > > > > > unnecessary > > > > > > > > > > > > > > > > builds from running. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Does anyone have any concerns with this > change > > or > > > > > > > > > suggestions? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Joe Evans > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > joseph.evans@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --000000000000745a0405a2292ecb--