singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <>
Subject [GitHub] [singa] joddiy commented on pull request #697: New Model Layer Operator API
Date Thu, 28 May 2020 04:40:02 GMT

joddiy commented on pull request #697:

   Hi, @dcslin, as you know, I need to develop and test the new API for onnx, so I have to
make sure the new implement of autograd and layers is ok. However, when I test the PR #697
, I have three types of issues of the ``:
   # Zero-value tensors
   For example, in the test case of `sum`, the value GPU tensor always is zero. But when I
remove the conv2d test case, the `sum` case will be fine. It seems the conv2d layer results
in the zero GPU tensor issue.
   The following cases have the same problem:
   - sum
   - onehot
   - tile
   - gather
   - split
   - slice
   - reduce_mean
   - reduce_sum
   - globalaveragepool
   - gemm
   - prelu
   - less_broadcast
   - xor_broadcast
   - reciprocal
   - xor
   - not
   - or
   - selu
   - prelu
   - min_3inputs
   - min
   - shape
   - squeeze
   - max_1inputs
   - max_3inputs
   - max
   - reshape
   - mul
   - transpose
   - unsqueeze
   an issue when I run the Conv2d with odd_padding. As you see, sometimes, I need to padding
zeros from only one direction, so I write this function:
   def handle_odd_pad_fwd(x, odd_padding):
       handle odd padding mode forward
           x, the input tensor
           odd_padding, the odd_padding
           tensor, the output
       x_tensor = tensor.from_raw_tensor(x)
       # (axis, left padding if True else right padding)
       flags = [(2, True), (2, False), (3, True), (3, False)]
       for (axis, left), pad in zip(flags, odd_padding):
           if pad == 0:
           zeros_shape = list(
           zeros_shape[axis] = pad
           zero_padding = np.zeros(zeros_shape).astype(np.float32)
           zero_padding = tensor.Tensor(device=x.device(), data=zero_padding)
           if left:
               x_tensor = tensor.concatenate((zero_padding, x_tensor), axis)
               x_tensor = tensor.concatenate((x_tensor, zero_padding), axis)
   But it seems, when I call this func, it'd be fine if I call only one or two times, however,
if I call it more times, it will report a error:
   > F0526 12:53:40.017063 15641 tensor_math_cuda.h:791] Check failed: status == CURAND_STATUS_SUCCESS
   I guess, the reason maybe it doesn't release the GPU memory in time?
   The following cases have the same problem:
   - conv2d
   - pooling2d
   - div_broadcast
   The third error msg is:
   > F0526 18:52:07.318809 21112 tensor_math_cuda.h:193] Error on line 193: CUDNN_STATUS_EXECUTION_FAILED
   And the following cases have the same problem:
   - SeparableConv2d
   - mul_broadcast
   - sub_broadcast
   - add_broadcast
   - greater_broadcast
   - or_broadcast
   - and_broadcast
   - negative
   - min_1inputs

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

View raw message