singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] Belegkarnil opened a new issue #681: AlexNet bacward shape missmatch + ReLu return a tuple
Date Mon, 20 Apr 2020 14:02:52 GMT

Belegkarnil opened a new issue #681:
URL: https://github.com/apache/singa/issues/681


   Hi,
   
   I have implemented AlexNet in singa but I obtain an error during the backward_and_update
instruction. I am using Singa 3.0.0.rc1 on cpu.
   
   This is my AlexNet implementation:
   `from singa import autograd
   from singa import module
   from singa import opt
   
   __all__ = ['AlexNet', 'alexnet']
   
   class AlexNet(module.Module):
   	def __init__(self, num_classes=1000):
   		super(AlexNet, self).__init__()
   		# 12 sur GPU donc 6 & 6
   		self.features1 = [
   			autograd.Conv2d(3,64,kernel_size=11,stride=4,padding=2),
   			autograd.ReLU(),
   			autograd.MaxPool2d(kernel_size=3, stride=2),
   			autograd.Conv2d(64,192,kernel_size=5,padding=2),
   			autograd.ReLU(),
   			autograd.MaxPool2d(kernel_size=3, stride=2),
   			autograd.Conv2d(192,384,kernel_size=3,padding=1),
   			autograd.ReLU(),
   			autograd.Conv2d(384, 256,kernel_size=3,padding=1),
   			autograd.ReLU()
   		]
   		self.features2 = [
   			autograd.Conv2d(256, 256,kernel_size=3,padding=1),
   			autograd.ReLU(),
   			autograd.MaxPool2d(kernel_size=3, stride=2)
   		]
   		self.avgpool = autograd.AvgPool2d(6, stride=1)
   		self.flatten = autograd.Flatten()
   		self.classifier = [
   			autograd.Dropout(),
   			autograd.Linear(256 * 6 * 6, 4096),
   			autograd.ReLU(),
   			autograd.Dropout(),
   			autograd.Linear(4096, 4096),
   			autograd.ReLU(),
   			autograd.Linear(4096, num_classes)
   		]
   		self.optimizer = opt.SGD(lr=0.001, momentum=0.9)
   	def loss(self, out, ty):
   		return autograd.softmax_cross_entropy(out, ty)
   	def optim(self, loss, dist_option, spars):
   		if dist_option == 'fp32':
   			self.optimizer.backward_and_update(loss)
   		elif dist_option == 'fp16':
   			self.optimizer.backward_and_update_half(loss)
   		elif dist_option == 'partialUpdate':
   			self.optimizer.backward_and_partial_update(loss)
   		elif dist_option == 'sparseTopK':
   			self.optimizer.backward_and_sparse_update(loss, topK=True, spars=spars)
   		elif dist_option == 'sparseThreshold':
   			self.optimizer.backward_and_sparse_update(loss, topK=False, spars=spars)
   	def forward(self, x):
   		for (i,layers) in enumerate([self.features1, self.features2, [ self.avgpool,self.flatten
 ] , self.classifier]):
   			for (j,fn) in enumerate(layers):
   				x = fn(x)
   				if(type(x) is tuple):# FIXME I have to do that because of a bug in Singa? (ReLU)
   					x = x[0]
   		return x
   
   def alexnet(**kwargs):
       return  AlexNet(**kwargs)
   `
   And I get : **AssertionError: ('shape mismatch', (9216, 4096), (256, 4096))**
   Which is my first linear layer : 256 * 6 * 6, 4096
   
   When I use my VGG16 implementation, I got a similar error :
   AssertionError: ('shape mismatch', (25088, 4096), (512, 4096))
   
   It seems that the backward operation does not map the correct shape to the corresponding
layer. 
   
   Moreover, the ReLu class return a 1-tuple containing a Tensor. Is it intended or is it
a bug?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message