Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mxnet.incubator.apache.org
MIME-Version: 1.0
From: Yao Wang <kevinthesunwy@gmail.com>
Date: Tue, 7 Nov 2017 14:46:53 -0800
Message-ID: <CAG6mzOOsMU=0B08Mup9V7a1gryxGqbn9ZRaGZ+d+OPugx-AQww@mail.gmail.com>
Subject: Data Pipeline Intermediate Representation in MXNet/NNVM
To: dev@mxnet.apache.org
Content-Type: multipart/alternative; boundary="f403045fbde0f1880c055d6c5844"
archived-at: Wed, 08 Nov 2017 00:00:20 -0000

--f403045fbde0f1880c055d6c5844
Content-Type: text/plain; charset="UTF-8"

Hi,

Tensorflow has a transform package https://github.com/tensorflow/transform
which is capable of export a data preprocessing pipeline to a tensorflow
graph, which can be incorporated into network graph. This package provides
a neat way to manage data pipeline together with network graph, since these
data process graph can be easily reused by other developers. Also I think
we can get some performance improvement by using computation graph for data
process rather than imperative processing for large data stream?

Currently in MXNet, if I want to do the similar thing, I need to pack the
code(most time python script) directly with network graph files. This
method has some issues:
1. Potential security issue. If I wrote the processing codes and I am the
only person use it, it's fine. However, if someone else wants to reuse it
in their application, they need to check the code to make sure there is no
security issue. It is not quite portable for reusing.

2. It is bind to specific language. Usually it's easier to develop deep
learning application using python, but if my production environment doesn't
have python environment, I need to either setup python environment or
rewrite this script with the language supported by my production
environment.

Any thought about supporting data pipeline IR in MXNet/NNVM?

--f403045fbde0f1880c055d6c5844--