From dev-return-2831-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Thu May 10 16:42:43 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BACA618063A for ; Thu, 10 May 2018 16:42:42 +0200 (CEST) Received: (qmail 72477 invoked by uid 500); 10 May 2018 14:42:41 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 72461 invoked by uid 99); 10 May 2018 14:42:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 May 2018 14:42:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id AA568C02DA for ; Thu, 10 May 2018 14:42:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.899 X-Spam-Level: * X-Spam-Status: No, score=1.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id CjWbUsDsVv8G for ; Thu, 10 May 2018 14:42:39 +0000 (UTC) Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 959835F254 for ; Thu, 10 May 2018 14:42:38 +0000 (UTC) Received: by mail-io0-f182.google.com with SMTP id z4-v6so3282623iof.5 for ; Thu, 10 May 2018 07:42:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=G00iCLnYK2M3ZJjFotngQNjjR7xW+/4Mo4TapGeFSlo=; b=fNxPHGukomxmRhbvsUOkvYb8ATd0tfcGWOe2kOMgrbQmYlLiw3cI3dM3Dlo9xTRGGU vJWRZekFDORlnSIV1j4OHRbgvmI7MNOvDbgsIWFsLO6QsaMf6BfN8QG/9P6yekg1W7fc JKEAfju/xCaz1Ox92HLU6pvRw9eFPbtzzj6JpUEPXEw2/cpOxDwpWlqpnVAeucSKJOfN aPkQ20q/57kxZaJdLuuMxEzkvkahIRqMwnutkPsupOL8NEbqdPQdp6fxlwVwTijLCqii mB6a77k0ZPmWr+wCHJa6ryRda4Og6z6UDyu5mB9IUXQRulClw5kCUzGousiHjZXCFWcc CT+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=G00iCLnYK2M3ZJjFotngQNjjR7xW+/4Mo4TapGeFSlo=; b=UssIdC3hOMGi+K5qXJVxyvLe6NrBVZ0pDBVQEOWdcgcDdrKVNdlCAwxkXH71+kxvEe QGr0858nKXRbJlfbgyO8wRXfbXKQBErgsm+ppVFSOZ3ziEV/IC+1G+B1VheNOakA7PRW wvddn2BLmSxh6xvFjLwrB4SW+eE0zyqF3O38xId1IzjPmRfpQPRnBMKDE3Hh+b5dhFss 9tP4qeKde49PQemyC1dF8i17ed/lEERxsUwKOJB2gqcQ8UwqNQmF7tIT6aGuXLxHmTFC VVsyzgi1/mHP5m9rtYJc9uw/77Ux7uomSZcjnSBkxmUTF8zpWznElh0jlYvrD5V0BYsT xGcg== X-Gm-Message-State: ALKqPwfORwD8btUwiaOWwm2TJgCcQD0OhcdK8e1jexxextN2XQUxbFHB B/BHY6rYRV7ymNcx5E3H4kiuMUdlO7pgqmDon+lbDJV6 X-Google-Smtp-Source: AB8JxZqz78+tUq/XTwWt0i+7AuarYFglMJ+gFQ59g5wzWt9ThZMyiKBSlIDEExbPR5sqbkG+if/WVwxDhwL7q9Nt9xk= X-Received: by 2002:a6b:998d:: with SMTP id b135-v6mr1893912ioe.122.1525963356846; Thu, 10 May 2018 07:42:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.192.165.242 with HTTP; Thu, 10 May 2018 07:42:36 -0700 (PDT) From: kellen sunderland Date: Thu, 10 May 2018 16:42:36 +0200 Message-ID: Subject: Parallel Inference Proposal To: dev@mxnet.incubator.apache.org Content-Type: multipart/alternative; boundary="000000000000d34266056bdb0794" --000000000000d34266056bdb0794 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello MXNet developers, I=E2=80=99ve recently been speaking with users who=E2=80=99d like to run pa= rallel inference requests with MXNet on their service. They=E2=80=99ll do this on GPUs, and= due to resource constraints, they=E2=80=99d like to do this without duplicating th= eir model=E2=80=99s weights in memory. They=E2=80=99d also like run inference = with a low degree of buffering/batching as latency is important. I=E2=80=99ve created= a wiki page with a small proposal that I hope will make running parallel inference a little easier. I=E2=80=99d like to discuss the proposal in this thread a= nd would particularly appreciate it if core devs could correct me if I=E2=80=99ve ma= de any incorrect assumptions in the doc. Proposal here: https://cwiki.apache.org/confluence/display/MXNET/Parallel+Inference+in+MXN= et If people are OK with the proposal I can open a Jira ticket, PR, etc. If people are curious about perf implications I can also do some benchmarking. Thanks in advance for the feedback, -Kellen --000000000000d34266056bdb0794--