mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sheng Zha <>
Subject [apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)
Date Fri, 13 Sep 2019 21:48:00 GMT
# Overview

The purpose of this RFC is to organize and present the roadmap towards 2.0. As 2.0 will be
a major release, changes that would break backward compatibility are permissible.

The proposed changes in this RFC are either collected from past roadmap discussions such as
#9686, or are based on various common issues from the past. This RFC organizes these changes
into self-contained projects to facilitate clear definition of project, captures the risks
and status quo to the best of our knowledge. To help navigate, the projects are further divided
into several high-level areas. Some of the listed projects are already in progress, and are
included to provide a clear overview.

The objectives of Apache MXNet 2.0 include:
- Improve expressiveness and usability of user-facing API.
- Improve expressiveness and usability of the technical stack for lower development cost and

In terms of frontend, this roadmap focuses mostly on Python-frontend since MXNet has been
taking a Python-first approach. The expectation with respect to other language bindings is
that they would evolve along with the backend evolution and make use of the improvements.
Given that breaking changes can occur, maintainers of different language bindings are expected
to participate in related interface definition discussions.

## N1. NumPy

NumPy has long been established as the standard math library in Python, the most prevalent
language for the deep learning community. With this library as the cornerstone, there are
now the largest ecosystem and community for scientific computing. The popularity of NumPy
comes from its flexibility and generality.

In #14253, the MXNet community reached consensus on moving towards a NumPy-compatible programing
experience and committed to a major endeavor on providing NumPy compatible operators.

The primary goal of the projects below is to provide the equivalent usability and expressiveness
of NumPy in MXNet to facilitate Deep Learning model development, which not only helps existing
deep learning practitioners but also provides people in the existing NumPy community with
a shortcut for getting started in Deep Learning. The efforts towards this goal would also
help a secondary goal, which is to enable the existing NumPy ecosystem to utilize GPUs and
accelerators to speed up large scale computation.

cc @apache/mxnet-committers 

### NumPy Operator Testing

1. adopt __array_function__ and numpy existing tests.
2. extend testing to GPU
3. investigate numpy testing strategies
4. decide correctness criteria for acceptance

### NumPy Operator performance profiling

1. Automatically profile the performance of NumPy operators

### NumPy operator coverage

1. improve operator until full NumPy coverage, with prioritization towards operators used
in the ecosystem and deep learning in general

Operator coverage as of 07/03/2019

|    module |     NumPy | deepNumPy |       jax |      cupy |
|        np |       603 |        89 |       445 |       321 |
|   ndarray |        71 |        32 |        71 |        56 |
|    random |        63 |         5 |        15 |        49 |
|    linalg |        31 |         2 |         8 |        15 |

### NumPy Extension Operator Reorganization and Renaming

1. consistent type usage for index input and return values from sort, topk #11031 #11134,
2. array creation operators with flexible dtype definition #12290. (dtype=None)
3. moving_mean/moving_var in batchnorm
4. consistent usage of axis vs dim
5. promote or deprecate contrib operators

### NumPy ndarray type extension

1. bfloat16 support (not in NumPy yet but useful for deep learning) (low priority — Intel)
2. boolean type support
3. complex (for FFT)

### NumPy ndarray boolean indexing

1. allow boolean masks in NumPy ndarray indexing by adding the operator, potentially through
extending op.where

### Hybridizable basic (and advanced) indexing


1. Allow operations such as y = x[1:3, 2, ...] to be hybridizable

Note: Preliminary work:

## Graph Enhancement and 3rdparty support

The objective of the following projects is to enable easier development of third-party extensions
without requiring changes to be checked in the MXNet project. Examples of such extensions
include third-party operator library and accelerators.

### Graph Partitioning for Dynamic Shape Operators

1. partition inside control flow operators (and all cached ops)
2. partition on operators with dynamic shapes for partial memory planning and caching.

### Improved Third-party Operator Support

1. allow registering custom operators by exposing C API (and frontend API) to register NNVM
op at runtime.
2. verify serialization, deserialization, and graph passes for graphs with these operators
are working properly.

### Improved Third-party Backend Support (subgraph property)

1. expose a graph pass for standard graph partitioning with back-end-specific criteria as
a C API and frontend API.

### Large tensor support by default

1. enable default support for tensor with int64 dimension sizes
2. make sure there’s no significant performance regression in operators

1. performance regression may happen in a subset of operators, which can disproportionally
affect certain models.
2. compatibility and silent behavior change.

Notes: in progress (RFC:

## API Changes

The objective of the following projects is to address the technical debts accumulated during
the development of MXNet 0.x and 1.x with respect to the API definition.

### C-API Clean-up

C-API is the foundational API in MXNet that all language bindings depend on.

1. use packed function for flexibility (and potentially efficiency through avoiding string
2. do not expose backend accelerator-specific types such as mkldnn::memory in C-API
3. do not rely on topological ordering for argument passing (#15362).
4. verification of thread-safety and performance for C API

1. backend integration may require refactoring or even redesign
2. existing use cases such as other frontend may be broken without substitute
3. feedback is scattered and we may miss the opportunity to change some APIs in 2.0

### Unify Executor

1. SymbolBlock equivalent in C/C++, unify the executor implementation for symbol/module and
the one for gluon blocks
2. migrate other versions of inference API

### Gradient of Gradient support

1. higher order gradient support for a subset of operators

1. large number of backward operators could introduce significant technical debt if not properly
2. ill-informed prioritization may result in usability issue (e.g. common GAN not supported)

### Autograd Extension

1. improve interface to support specifying intermediate output grad nodes
2. improve interface for better usability. (retain_graph → something not involving graph)
3. update graph pass for correctness

### NNVM-backend Operator Interface Changes

1. support more than one temporary spaces
2. split forward shape/type inference and reverse shape/type inference for better error messaging.
3. deferred initialization removal (or improve error/info message)
4. accompanying operator implementation changes

1. some changes may make operator implementation less error-prone while less flexible, and
thus require some reworking.

## Gluon 2.0

Since the introduction of the Gluon API, it has superceded other API for model development
such as symbolic API and model API. Conceptually, Gluon is the first attempt in the deep learning
community to unify the flexibility of imperative programming with the performance benefits
of symbolic programming, through trace-based just-in-time compilation.

The objectives of the following projects are:
- address usability issue as a result of the divergence in the behavior of NDArray and Symbol.
- extend the JIT to improve the coverage of hybridization.
- introduce new functionality to facilitate more areas of research such as Baysian methods
and AutoML.
- improve the usability and performance of the utility in Gluon.

### Unifying symbolic and imperative mode for tensor library

1. unify the operator implementation and behaviors of symbolic and imperative execution modes
2. allow naming for ndarray similar to symbol
3. address the necessary changes in shape/type inference.

### Unifying Block and HybridBlock

1. move hybridization logic to a JIT decorator
2. extend parameter management to Block
3. user-friendly warning for native control flow in JIT code.

### Gluon Block Enhancement

1. inspection of graph internals similar to monitor for Module ([PR 15839](
2. support additional types in argument such as dict, kwargs, None
3. fused parameters and gradients respectively
4. register custom parameter

### Enable Symbolic Shape (& Dtype) for Array Creation in NNVM-backend

1. allow flexible creation of array based on shapes of other arrays that are only known at
2. add constant symbol type as the return value of symbol.shape (?)
3. support constant symbol as operator arguments (?)
4. constant folding for constant symbols

### Gluon Distributions Module

1. sampling and pdf definition for distributions. Distribution
2. wrap operators into more usable classes.
3. reproducible global seed

### Gluon Metrics Module

1. address usability and performance issues in mxnet.metric using hybridizable NumPy op

### Gluon Optimizer Module

1. API changes such as consistent weight decay (#9881), change default value to not apply
wd on bias terms (#11953)
2. hybridizable optimizers
3. new optimizers (#9182)

### Gluon Data API Extension and Fixes

1. address diverging interfaces and remove transform= constructor arg (#11141).
2. reorganize io/image modules and provide data loader instead.
3. lowering dataloader to backend for efficiency (#13593)
4. shared memory propagation?

### Gluon Estimator Extension for Experimenting Utilities


1. logging of configuration (DeepNLU), state, and performance for checkpointing for easier
2. pre-defined estimators for common problems

### Gluon Estimator Refactoring for Examples and Tutorials

1. modularize and refactor unstructured scripts and examples into estimator class utilities

### Gluon Distributed Training Usability Enhancement

1. more flexibility for communication with kvstore UDFs
2. add distribution strategies to estimator
3. plugin for communication backends (horovod, byteps, parameter server) for data parallel
4. data sharding/sampling/streaming enhancement for distributed training

### NNVM-Graph optimization

1. fix mirror for memory optimization (Bojian)

## Documentation

Documentation is the most important factor for new adoption of a library. The following projects
aim to:
- address the usability and discoverability issues in the current MXNet website
- improve the quality of documentation to make it correct, clear, and concise.
- help adoption of the changes in MXNet 2.0 from existing users.

### MXNet 2.0 Migration Guide

1. document high-level mapping from old functionality to new API for data pipeline, modeling,
optimization, training loop, metric, inspection and logging, debugging.

1. parallel development of the doc may result in outdated doc.
2. auto doc verification is needed.

### MXNet 2.0 Developer Guide

1. carefully document the design and contribution guide for features with low entry bar such
as operator, gluon block, doc, optimizer, metric, examples and tutorials.
2. clear and up-to-date system design overview.
3. clear roadmap

### Adopt as official website


1. infrastructure change for new doc build
2. merge into master with [](
3. improve load time and browsing experience
4. CDN in popular region such as China, with automated validation and testing.


## Profiling and Debugging

Profiling and debugging is a common step in the development of deep learning models, and proper
tools can help significantly improve developer's productivity. The objective of these projects
is to provide such tools to make it easier to discover issues in correctedness and performance
of models.

### Memory Profiler

1. memory profiler logging support in backend
2. automatic array naming tool based on scope
3. tree-map visualization tool for inspecting profiler dump

### Enhanced Debugging Tool

1. Enable user-specified error handling
2. Improve error message
3. Stacktrace inspection in debug API
4. Automatic error reporting tool
5. Runtime API for turning off asynchronous execution

## Advanced Operators

The objective of these projects are to extend the tensor library and operators for better
performance and for advanced use.

### Strided ndarray support

1. support strided array in a subset of operators
2. support auto-transpose of strided array in graph pass and executor

### Ragged ndarray and operators

1. introduce ragged (variable length) tensor as 1st class tensor. Support zero-copy from RaggedNDArray
to NDArray when no dimension is ragged.
2. Load balancing strategy for operators that take RaggedNDArray as input
3. cover operators for NLP applications (RNN, transformer)

### Improved Sparse Support

1. sparse format and operator support
2. scipy coverage
3. operators for graph neural-networks (e.g. ops in minigun)

Minimum support:

* format: csr,
* zerocopy to DLPack
* integration with minigun kernels

Next-level support:

* format: coo and block sparse.

## Building and Configuration

### CMake improvement and Makefile deprecation

1. reimplement CMakeLists for DMLC dependencies
2. reimplement CMakeLists for MXNet to support 1) building best performing binary in any platform
2) building portable binary distribution for pip

### MXNet Configurator

1. drop environment variables and centralize them as config.
2. define functionalities that support runtime-switch (candidates: memory pool, engine, worker
thread pools) and expose frontend API
3. allow saving and loading of mxnet system config

## Advanced training and deployment

### Automatic Quantization and Quantized Training for NumPy

1. automatic quantization based on heuristic (or learning)
2. BMXNet

Dependency: N1-N5

### Mobile and edge-device deployment

1. replace amalgamation with more user-friendly function (TF-lite equivalent).
2. tutorial and example
3. metal support without ONNX

## Performance

### MXNet Execution Overhead


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message