From dev-return-7026-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Wed Dec 11 05:35:37 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EBA6E180638 for ; Wed, 11 Dec 2019 06:35:36 +0100 (CET) Received: (qmail 36120 invoked by uid 500); 11 Dec 2019 05:35:35 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 36094 invoked by uid 99); 11 Dec 2019 05:35:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2019 05:35:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BB3CEC055C for ; Wed, 11 Dec 2019 05:35:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -9.8 X-Spam-Level: X-Spam-Status: No, score=-9.8 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_SPF_WL=-7.5] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=amazon.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 7_lYpu4BxJcI for ; Wed, 11 Dec 2019 05:35:33 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=52.95.49.90; helo=smtp-fw-6002.amazon.com; envelope-from=prvs=24154b626=sskalic@amazon.com; receiver= Received: from smtp-fw-6002.amazon.com (smtp-fw-6002.amazon.com [52.95.49.90]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id D20BC7DDAC for ; Wed, 11 Dec 2019 05:35:32 +0000 (UTC) X-ASF-DKIM-Sig: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1576042533; x=1607578533; h=from:to:cc:subject:date:message-id:mime-version; bh=9GyCvBHtpTUb0W6qs56+WkFEIFD36zZsJsBnxDccQ4A=; b=bJsFDIFZ4PgduUaz7aR8BJWOuTYYnui2M4z8PdNLtyk+4f+Gz+yMSjTc DY5aIH56N0imi85XECykamxREV7YY26IiZVDB0sNHRixrRpsPJBgIPh9g Q5CdNgccQFOX4h8UrRXYZ62wiLxIfO1tCxy0zOaBRQNGKcpfMrhYml2oq k=; IronPort-SDR: jDHSTKm9o9/MJJfyNcUEpaZ7GHpsg/idiDanP2aG654Wvw6zbU0vEZIwtV8ctKzIhcBQZGbovK 8appOOM6lhpw== X-IronPort-AV: E=Sophos;i="5.69,301,1571702400"; d="scan'208,217";a="7112752" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1e-97fdccfd.us-east-1.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 11 Dec 2019 05:35:32 +0000 Received: from EX13MTAUWA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-97fdccfd.us-east-1.amazon.com (Postfix) with ESMTPS id 3A119A242F for ; Wed, 11 Dec 2019 05:35:30 +0000 (UTC) Received: from EX13D06UWA002.ant.amazon.com (10.43.160.143) by EX13MTAUWA001.ant.amazon.com (10.43.160.118) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 11 Dec 2019 05:35:30 +0000 Received: from EX13D23UWA003.ant.amazon.com (10.43.160.194) by EX13D06UWA002.ant.amazon.com (10.43.160.143) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Wed, 11 Dec 2019 05:35:30 +0000 Received: from EX13D23UWA003.ant.amazon.com ([10.43.160.194]) by EX13D23UWA003.ant.amazon.com ([10.43.160.194]) with mapi id 15.00.1367.000; Wed, 11 Dec 2019 05:35:30 +0000 From: "Skalicky, Sam" To: "dev@mxnet.incubator.apache.org" CC: "Keshavan, Arjuna" , "Harish, Nihal" Subject: Performance regression from removing libiomp5.so Thread-Topic: Performance regression from removing libiomp5.so Thread-Index: AQHVr+TMonGrmxdul0e72aNGW7oYCw== Date: Wed, 11 Dec 2019 05:35:30 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.43.161.179] Content-Type: multipart/alternative; boundary="_000_B522773F66294B148508EE306F571A4Aamazoncom_" MIME-Version: 1.0 --_000_B522773F66294B148508EE306F571A4Aamazoncom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi MXNet community, I would like to bring your attention to the performance regression that was= found [1] between 1.5.1 and 1.6.0 due to removing the libiomp5.so library = due to licensing issues. This change was made since this library has a cate= gory x license [2] that is not compatible with the MXNet Apache license/dis= tribution. We found that using OpenBLAS instead of MKL BLAS caused a regression from 1= 500 samples/sec to 1300 samples/sec a 13.3% regression in training speed fo= r a resnet18 training benchmark on a C5.18xlarge EC2 instance (with 72 core= s). Rebuilding with MKL BLAS showed an increase in performance to 1600 samp= les/sec in the 1.6.0 branch. Please provide your feedback on the licensing issue (are there any work-aro= unds) and the tradeoff in performance (is the benefit worth trying to inclu= de back into MXNet builds). Thanks to the efforts of the following folks for working on this issue (in = no particular order): Patric Zhao Amol Lele Tao Lv A Pedro Larroy Nihal Harish Chai Bapat Arjuna Keshavan Rong Zhang Thanks! Sam [1] https://github.com/apache/incubator-mxnet/issues/16891 [2] https://www.apache.org/legal/resolved.html#category-x --_000_B522773F66294B148508EE306F571A4Aamazoncom_--