From dev-return-6958-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Tue Dec 3 18:19:09 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 14B5A180629 for ; Tue, 3 Dec 2019 19:19:08 +0100 (CET) Received: (qmail 73957 invoked by uid 500); 3 Dec 2019 18:19:08 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 73944 invoked by uid 99); 3 Dec 2019 18:19:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Dec 2019 18:19:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 78348180989 for ; Tue, 3 Dec 2019 18:19:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id xyHex6muoCXo for ; Tue, 3 Dec 2019 18:19:06 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::22a; helo=mail-lj1-x22a.google.com; envelope-from=pedro.larroy.lists@gmail.com; receiver= Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id B77B97DC1D for ; Tue, 3 Dec 2019 18:19:05 +0000 (UTC) Received: by mail-lj1-x22a.google.com with SMTP id j6so4965156lja.2 for ; Tue, 03 Dec 2019 10:19:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=aczlTADcXD4XCJdHD8G3/j4bCMGLhPtjhGlUamQ0rsc=; b=jRq2vrCD9Bfj/mDdxWwxLzZ+GfRcYKhglNbAJTN/ib39lVWvBvhaMiOB2/ASaZ+nsc qRPK/48ZXBn5xl9anCTh8Xvudgdr3hFSb3+s729b8tWLaETh79w+M3uS+NvNltuKMs12 8GZdSur8PSBhxrMQaINhWkYNIpkWjmO+aJaS5QRBeNl/FZMNysnb7hoXeMN+UM72+0sd 4kMAx4Lw6zgn6gHuoF3S5F4Iyfna1K4r2pNHOhPFQgq3pyicUtbb9WyICwgxIjQN3XA4 cpiXkvorvY7V6oY+sBlyRfKs+HsHjcEvWfjdzkGIggsJiDiuVqNAaS70CBl0LdgI3MN3 9qwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=aczlTADcXD4XCJdHD8G3/j4bCMGLhPtjhGlUamQ0rsc=; b=TylrWrVQ5YzkaJ6KplRFx3WPuE/tReX2m3DoDbhtpb6AUl+78rKKxz8/2etf/KMIqW Bce16YsXJgHTqhUwUbeBooGU50xT9JbpKYaH1uRzhfcEFVJF9bztQTOIKvwo6jphNrXj 9CK1ynOSzw1N0BjUsXWevf4/QpFxfNYqS18UMZBxlwRdzeQBF2YPgplYBxwIamum5tA2 uUPIRsD3Cefb+duOj2B7EEh2M1GMP3F4rtvHY9b4HHzbgsRg2goRZNfeSTsz3NSnblmS DKuwUdLjKn+0OYzIi2Vl8yub0yZ/a3i+khCfrU+cU9ibezXjADG6qm93t70SqM06ZnPX 3rTQ== X-Gm-Message-State: APjAAAUcdqliOPboIykntfmWbIHxY/bhPFcrKMRpscnfOTGLnd5r6XE7 0UYSsY3SpxRO/N35iilejuNGVDg9xRQ2fslrP+KOkyGsPlc= X-Google-Smtp-Source: APXvYqwVXduqE1d3DfevD0mvIS/G/hkrdcYH4T8PXbMoaeiREpET3DZjk9hn4kuHy+R4duJiKqKPq2ixdYqpktipWZU= X-Received: by 2002:a2e:894b:: with SMTP id b11mr3439514ljk.118.1575397145009; Tue, 03 Dec 2019 10:19:05 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Pedro Larroy Date: Tue, 3 Dec 2019 10:18:45 -0800 Message-ID: Subject: Re: CI Update To: dev@mxnet.incubator.apache.org Content-Type: multipart/alternative; boundary="00000000000035619e0598d0bbab" --00000000000035619e0598d0bbab Content-Type: text/plain; charset="UTF-8" Also please take note that there's a stage building TVM which is executing compilation serially and takes a lot of time which impacts CI turnaround time: https://github.com/apache/incubator-mxnet/issues/16962 Pedro On Tue, Dec 3, 2019 at 9:49 AM Pedro Larroy wrote: > Hi MXNet community. We are in the process of updating the base AMIs for CI > with an updated CUDA driver to fix the CI blockage. > > We would need help from the community to diagnose some of the build errors > which don't seem related to the infrastructure. > > I have observed this build failure with tvm when not installing the cuda > driver in the container: > > > https://pastebin.com/bQA0W2U4 > > centos gpu builds and tests seem to run with the updated AMI and changes > to the container. > > > Thanks. > > > On Mon, Dec 2, 2019 at 12:11 PM Pedro Larroy > wrote: > >> Small update about CI, which is blocked. >> >> Seems there's a nvidia driver compatibility problem in the base AMI that >> is running in GPU instances and the nvidia docker images that we use for >> building and testing. >> >> We are working on providing a fix by updating the base images as doesn't >> seem to be easy to fix by just changing the container. >> >> Thanks. >> >> Pedro. >> > --00000000000035619e0598d0bbab--