mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sam Skalicky <notificati...@github.com>
Subject Re: [apache/incubator-mxnet] [RFC] MXNet external operators (#18904)
Date Wed, 19 Aug 2020 08:41:15 GMT
Hey folks,

I was playing around and realized we can load anything we want following the example, not
just operators. For example, I wrote this little function to dump out the registered operators
and map the aliases:
```
#include "operator_common.h"

extern "C" int listOps() {
  // get op registry
  ::dmlc::Registry<::nnvm::Op>* reg = ::dmlc::Registry<::nnvm::Op>::Get();
  // get list of registered op names
  std::vector<std::string> ops = reg->ListAllNames();

  // create inverse map of Op to name (to find aliases)
  std::map<const ::nnvm::Op*,std::vector<std::string> > op_map;
  for(auto &name : ops) {
    const ::nnvm::Op* op = reg->Find(name);
    if(op_map.count(op) > 0) {
        if(name.compare(op->name) != 0)
          op_map[op].push_back(name);
    } else {
        op_map[op]={};
        if(name.compare(op->name) != 0)
          op_map[op].push_back(name);
    }
  }

  // print out the op mapping
  for(auto &kv : op_map) {
    std::cout << kv.first->name << ", ";
    for(auto &n : kv.second)
      std::cout << n << ", ";
    std::cout << std::endl;
  }
  
  return 0;
}
```
I compiled this code with MXNet to get an object file, and then built that into a shared library:
```
g++ -shared -fPIC -std=c++11 init_lib.cc op_info.cc.o ../external_ops/src/lib_api.cc -o libop_info.so
-I../external_ops/include -L../external_ops/build -lmxnet
```
Then, with this Python code I can load my library into MXNet and at the same time get a handle
to the library directly and call my function (now with access to the libmxnet.so):
```
import os
import ctypes
import mxnet as mx
mx.library.load(os.path.abspath('libop_info.so'))
libop_info = ctypes.CDLL('libop_info.so')
libop_info.listOps()
```
I was able to get this list of operators out: https://docs.google.com/spreadsheets/d/1BK2UBJW7NFicriYGRpFwZHFi1HYqXwPTTzaY4jSKaWs/edit#gid=0

So, should we consider making this proposal more than just "external operators"? Maybe "external
components" that could be anything: operators, subgraph properties, utilities, etc....Thoughts?

Sam

-- 
You are receiving this because your review was requested.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/pull/18904#issuecomment-675956750
Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message