mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taliesin Beynon <talies...@wolfram.com.INVALID>
Subject trouble with foreach operator in conjunction with multiple GPUs
Date Wed, 28 Nov 2018 14:11:39 GMT
Hello fellow MXNetters

We've seen that the subgraph execution mechanism that is used to run things like the foreach
operator causes MXExecutorForward to block, instead of just issuing the ops in the normal
asynchronous way (https://github.com/apache/incubator-mxnet/blob/212364b0cba28aeda989378f6e630f7a61749bf3/src/executor/graph_executor.cc#L1352).
On its own this is a surprising fact that can lead to some issues if you're not expecting
it, like your time being spent in MXExecutorForward instead of WaitAll / WaitRead . Is there
a reason that this process isn't just automatically done on a separate thread for you? Is
it to ensure that subsequent ops on the original thread are correctly serialized wrt the ops
produced by the foreach? 

More importantly, this has the unfortunate implication that if you are using multi-device
parallelism with foreach, by just looping over your executors and calling Forward on them,
you will inadvertently serialize much of the computation: you can't call Forward on the second
executor until Forward on the first executor has returned, and the foreach causes that first
Forward call to block until the forward pass is (mostly) done!

So it kills multi-device parallelism unless one starts making thread pools so that the one
can 'unblock' Forward (and probably the subsequent Backward) and have each device's Forward
being run in a separate thread. 

Is this intended? Are we missing something about how you are supposed to use subgraphs in
conjunction with multi-device parallelism? It seems like a weakness in the current design
of subgraph execution. It also appears that the python API doesn't have any strategy to deal
with this issue, as you can see on https://github.com/apache/incubator-mxnet/blob/2276bb0e30b1fe601eb288cb4f1b673484892d4b/python/mxnet/executor_manager.py#L281,
it's not making separate threads or anything there.

Thanks!
Tali + Sebastian
Mime
View raw message