mxnet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] ThomasDelteil commented on issue #9974: DataLoader with workers not compatible with ImageRecordDataset
Date Mon, 26 Mar 2018 23:46:22 GMT
ThomasDelteil commented on issue #9974: DataLoader with workers not compatible with ImageRecordDataset
URL: https://github.com/apache/incubator-mxnet/issues/9974#issuecomment-376347564
 
 
   hot-fix for this problem: use at your own risk:
   ```
   import mxnet as mx
   from mxnet import gluon
   from mxnet.gluon.data import RecordFileDataset
   from mxnet.gluon.data.dataloader import DataLoader
   from mxnet import recordio
   
   # We keep the filename as an attribute
   # So that we can open a new handle per process
   # in the dataloader
   
   def __init__new(self, filename):
       self._filename = filename
       self.reinitialize()
       
   def reinitialize(self):
       idx_file = os.path.splitext(self._filename)[0] + '.idx'
       self._record = recordio.MXIndexedRecordIO(idx_file, self._filename, 'r')
       
   RecordFileDataset.reinitialize = reinitialize
   RecordFileDataset.__init__ = __init__new
   
   # We modify the dataloader worker_loop to reinit the dataset if possible
   # And then call to the original worker_loop
   
   gluon.data.dataloader.worker_loop_old = gluon.data.dataloader.worker_loop
   
   def worker_loop_new(dataset, key_queue, data_queue, batchify_fn):
       if 'reinitialize' in dir(dataset):
           dataset.reinitialize()
       gluon.data.dataloader.worker_loop_old(dataset, key_queue, data_queue, batchify_fn)
   
   gluon.data.dataloader.worker_loop = worker_loop_new
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message