From dev-return-7649-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Mon Jun 1 18:16:52 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 550C51804BB for ; Mon, 1 Jun 2020 20:16:52 +0200 (CEST) Received: (qmail 80047 invoked by uid 500); 1 Jun 2020 18:16:51 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 80035 invoked by uid 99); 1 Jun 2020 18:16:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2020 18:16:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id AA004180C30 for ; Mon, 1 Jun 2020 18:16:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.002 X-Spam-Level: X-Spam-Status: No, score=0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id qIjaCjthNDRu for ; Mon, 1 Jun 2020 18:16:49 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.210.54; helo=mail-ot1-f54.google.com; envelope-from=mnnaveen@gmail.com; receiver= Received: from mail-ot1-f54.google.com (mail-ot1-f54.google.com [209.85.210.54]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 40368BBEB7 for ; Mon, 1 Jun 2020 18:16:49 +0000 (UTC) Received: by mail-ot1-f54.google.com with SMTP id u23so8747181otq.10 for ; Mon, 01 Jun 2020 11:16:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=u1V2bLkDvv8bwONynKFyopjKW6Mq+c9h155IVdRrBsw=; b=TVIV3IBYy3M8gKIaanlNid/sHdE6MA3x33mFJH9RDsb1pe15JqxUwSDFxcpduT+CUy lCoDlApJrOuTbtUmkonh+NXffvFnAvmqyZqa4WMo5bG2cjqJF1Nj/Jd0ckNaVBLACyAK RN4Yd6p/O5qTcgT4vHo/C66IiS1FHbX3RSY+J/3FJ9HwL4V3TLaV55pkjg2mgfl0CmF4 kak7Q/Z8JxDnlkk7x+0Y+mxqHXgy5I+/QW2a3unELfA/dGDqlHZVOIlsm2pFfO2KZ3FA oiMdtuHy/cQVahL+TJ531JIJwgO4R3ciQYS0CdDRnQY3kzjTsvDpa3sxaIrSTcUGa11Q /9Ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=u1V2bLkDvv8bwONynKFyopjKW6Mq+c9h155IVdRrBsw=; b=Mu7AJ9zDW6oVwrQEPH7lUdAV8BgS1M28l+0Chv8Qhoj02wtg+b3Ub0UlA8tAwKuAy+ CccN9lrIHUupXxGqcwMGSuJC54XQR2j33djoqi7xkf7liLqg9+DpOJxppiLrJlj32Ek/ aJOf5xgQ5qRf8z4F4rbdNUHXJFVGzJyQCHQt6nhXdO0QgqL42hEhFIsAW2a+pUF4fxpA X5dHyJC7VHplzIgUoT/5hEnMlMD5zQJ7EDDVaYb60ukH1/+/XZ9j8lSP+hKUiwhrBUhp oZP6TqBbfwjtJubgt8Bq7LaHGG+pD/8FRyjrVGsbuVGpX9AiPjFtdYHONgvgcuOFGI+V NsBA== X-Gm-Message-State: AOAM5323K3L9x7AsUnhHOvyel1FMOqGb2w3CM2dYgJS7J0omuZKQiZ9P EIUFLvuYiGNfM4lY56xVzFn903EpHTAmqvXr27lodRYn X-Google-Smtp-Source: ABdhPJwdRwMfclUld196LVUGRoFajzsyzqnpGNH0SKYW9Bt9qd1Hfu2kXzE55K6gE6zFwwVnfvYN90keYEmR0l5DcNo= X-Received: by 2002:a9d:906:: with SMTP id 6mr16933676otp.354.1591035408333; Mon, 01 Jun 2020 11:16:48 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Naveen Swamy Date: Mon, 1 Jun 2020 11:16:12 -0700 Message-ID: Subject: Re: Profiler Broken? To: dev@mxnet.incubator.apache.org Content-Type: multipart/alternative; boundary="00000000000056c36705a709cc6b" --00000000000056c36705a709cc6b Content-Type: text/plain; charset="UTF-8" FYI, It works with 1.6.0, the output json file was too large for chrome to handle and it silently failed. I reduced the number of iterations to profile to create a smaller file, anything above 500 MB seems to fail. Thanks Sheng for the pointer and Anirudh for the help. On Thu, May 28, 2020 at 1:07 PM Pedro Larroy wrote: > Yes the profiler seems to be broken / has some concurrency issues. I have > seen corrupted profile results. > > On Thu, May 28, 2020 at 12:30 PM Naveen Swamy wrote: > > > I am attempting to profile one of our models, I used the profiler.state > to > > run/stop in code and also used the environment variables to autostart the > > profiler. It creates a 600MB json file, however when I view in chrome > > tracing it comes out to be blank screen (loading seems to be fine, didn't > > get any errors) > > > > Wondering if anyone has recently tried or if aware of profiler being > > broken? > > > > ENVIRON: Ubuntu 18.04 > > MXNet : mxnet-cu101mkl > > Deep Learning AMI (Ubuntu 18.04) Version 29.0 (ami-043f9aeaf108ebc37) > > > > Thanks, Naveen > > > --00000000000056c36705a709cc6b--