mxnet-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-mxnet] samskalicky opened a new issue #19655: Race condition between loading a model and accessing tensor data
Date Fri, 11 Dec 2020 01:24:21 GMT

samskalicky opened a new issue #19655:
URL: https://github.com/apache/incubator-mxnet/issues/19655


   ## Description
   Theres a race condition between loading a model and the model's params being available
by a backend run with `optimize_for` like:
   ```
   block = SymbolBlock.imports(symbol_file, input_names, param_file, mxnet.cpu())
   block.optimize_for(inputs, backend='myBackend')
   ```
   Where the data in the ndarrays have garbage data in the backend.
   
   ### Fix
   We need to loop over the NDArrays for args/aux and call `wait_to_read` on each to ensure
any previous operations pushed to the engine are complete:
   https://github.com/apache/incubator-mxnet/blob/master/src/c_api/c_api_symbolic.cc#L1309-L1310
   
   ### Workaround
   In the meantime we can simply call `mx.nd.waitall()` between loading the model and `optimize_for`
   ```
   block = SymbolBlock.imports(symbol_file, input_names, param_file, mxnet.cpu())
   mx.nd.waitall()
   block.optimize_for(inputs, backend='myBackend')
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org


Mime
View raw message