mxnet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] piiswrong opened a new pull request #8413: [WIP] Proof of concept multi processing
Date Thu, 01 Jan 1970 00:00:00 GMT
piiswrong opened a new pull request #8413: [WIP] Proof of concept multi processing
URL: https://github.com/apache/incubator-mxnet/pull/8413
 
 
   This is a proof of concept multi processing IO refactor. 
   only works for linux/mac.
   
   It can be tested with
   ```
   import time
   import numpy as np
   import multiprocessing as mp
   import mxnet as mx
   from mxnet import gluon as gl
   
   
   class ImageDataset(gl.data.Dataset):
   	def __init__(self, N):
   		self.N = N
   
   	def __len__(self):
   		return self.N
   	
   	def __getitem__(self, key):
   		img = mx.image.imread('data/test_images/ILSVRC2012_val_00000001.JPEG')
   		return mx.image.center_crop(img, (224, 224))[0]
   
   
   if __name__ == '__main__':
   	a = mx.nd.zeros((10,)) # trigger engine
   	mp.set_start_method('fork')
   	N = 10000
   	M = 10
   	BS = 100
   
   	dataset = ImageDataset(N)
   	data_loader = gl.data.DataLoader(dataset, BS, shuffle=True, num_workers=M)
   
   	tic = time.time()
   
   	for batch in data_loader:
   		print(batch.shape)
   
   	print(N/(time.time() - tic))
   ```
   @winstywang @yajiedesign 
   
   This also fixes the problem that mxnet doesn't work with fork() @andreaolgiati 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message