singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] joddiy commented on pull request #697: New Model Layer Operator API
Date Thu, 28 May 2020 04:40:02 GMT

joddiy commented on pull request #697:
URL: https://github.com/apache/singa/pull/697#issuecomment-635095451


   Hi, @dcslin, as you know, I need to develop and test the new API for onnx, so I have to
make sure the new implement of autograd and layers is ok. However, when I test the PR #697
, I have three types of issues of the `test_operation.py`:
   
   # Zero-value tensors
   
   For example, in the test case of `sum`, the value GPU tensor always is zero. But when I
remove the conv2d test case, the `sum` case will be fine. It seems the conv2d layer results
in the zero GPU tensor issue.
   
   The following cases have the same problem:
   - sum
   - onehot
   - tile
   - gather
   - split
   - slice
   - reduce_mean
   - reduce_sum
   - globalaveragepool
   - gemm
   - prelu
   - less_broadcast
   - xor_broadcast
   - reciprocal
   - xor
   - not
   - or
   - selu
   - prelu
   - min_3inputs
   - min
   - shape
   - squeeze
   - max_1inputs
   - max_3inputs
   - max
   - reshape
   - mul
   - transpose
   - unsqueeze
   
   # CURAND_STATUS_LAUNCH_FAILURE
   
   an issue when I run the Conv2d with odd_padding. As you see, sometimes, I need to padding
zeros from only one direction, so I write this function:
   ```py
   def handle_odd_pad_fwd(x, odd_padding):
       """
       handle odd padding mode forward
       Args:
           x, the input tensor
           odd_padding, the odd_padding
       Returns: 
           tensor, the output
       """
       x_tensor = tensor.from_raw_tensor(x)
       # (axis, left padding if True else right padding)
       flags = [(2, True), (2, False), (3, True), (3, False)]
       for (axis, left), pad in zip(flags, odd_padding):
           if pad == 0:
               continue
           zeros_shape = list(x_tensor.data.shape())
           zeros_shape[axis] = pad
           zero_padding = np.zeros(zeros_shape).astype(np.float32)
           zero_padding = tensor.Tensor(device=x.device(), data=zero_padding)
           if left:
               x_tensor = tensor.concatenate((zero_padding, x_tensor), axis)
           else:
               x_tensor = tensor.concatenate((x_tensor, zero_padding), axis)
       return x_tensor.data
   ```
   
   But it seems, when I call this func, it'd be fine if I call only one or two times, however,
if I call it more times, it will report a error:
   
   > F0526 12:53:40.017063 15641 tensor_math_cuda.h:791] Check failed: status == CURAND_STATUS_SUCCESS
(201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE
   
   I guess, the reason maybe it doesn't release the GPU memory in time?
   
   The following cases have the same problem:
   - conv2d
   - pooling2d
   - div_broadcast
   
   # CUDNN_STATUS_EXECUTION_FAILED
   The third error msg is:
   > F0526 18:52:07.318809 21112 tensor_math_cuda.h:193] Error on line 193: CUDNN_STATUS_EXECUTION_FAILED
   
   And the following cases have the same problem:
   - SeparableConv2d
   - mul_broadcast
   - sub_broadcast
   - add_broadcast
   - greater_broadcast
   - or_broadcast
   - and_broadcast
   - negative
   - min_1inputs


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message