singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] chrishkchris opened a new issue #744: Could we override the test precision atot of onnx test case, atot=1e-7 is too tough?
Date Fri, 19 Jun 2020 03:35:01 GMT

chrishkchris opened a new issue #744:
URL: https://github.com/apache/singa/issues/744


   The default of atot for onnx "model" is here
   https://github.com/onnx/onnx/blob/master/onnx/backend/test/case/model/__init__.py#L34
   
   The default of atot for onnx "node" is here
   https://github.com/onnx/onnx/blob/master/onnx/backend/test/case/node/__init__.py#L148
   
   Is that possible to override these setting in our test?
   
   Currently when run our onnx_test_backend.py, it all reports "Not equal to tolerance rtol=0.001,
atol=1e-07"
   
   I think absolute value error atol=1e-07 is too tough, it is too difficult to pass?
   
   ```
   root@d05828f767ee:~/dcsysh/singa/test/python# python3 test_onnx_backend.py
   ss............................ssssssssssssssssssssssssssssssss................ssss..ss..ss......FFFF..ssssssssss..ssssssssssssssss............ssss....................ssssssss........................ssss....ssssssssssssssssssssssssss........ssssssssssssssssssssssssssss..........ssssssssssssss......FsFsssssssssssssssss..................ssss....ssssss..ssss..........ss.s............ssss....ssssssssssssssssssss........ssssssssssss..............ssssssssssssssssssssssss......ss......ssss..ss........FFFF..ssssssssss............ssssssssssssssssssssssssss......ssss....ssssssssssssssssss..................................ss........ssssssssssssssss....ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss................ssssssssssssssssssssssssssssssss................ssssssssssssssssss....................ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss..........ssss......ssss........ssss..............ss..........................................ssssssssssss....................ssssssssssssssssssss....ssssssssssssssssssssssssssssss................ssssss................
   ======================================================================
   FAIL: test_averagepool_2d_same_lower_cpu (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 189 / 3072 (6.15%)
   Max absolute difference: 1.3230393
   Max relative difference: 3.
    x: array([[[[ 1.764052e+00,  1.082105e+00,  6.894476e-01, ...,
              1.501069e+00,  8.121531e-01,  2.665550e-01],
            [ 4.381333e-01, -1.760931e-01, -2.374533e-01, ...,...
    y: array([[[[ 4.410131e-01,  5.410524e-01,  3.447238e-01, ...,
              7.505345e-01,  4.060766e-01,  1.332775e-01],
            [ 2.190667e-01, -1.760931e-01, -2.374533e-01, ...,...
   
   ======================================================================
   FAIL: test_averagepool_2d_same_lower_cuda (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 189 / 3072 (6.15%)
   Max absolute difference: 1.3230393
   Max relative difference: 3.
    x: array([[[[ 1.764052e+00,  1.082105e+00,  6.894476e-01, ...,
              1.501069e+00,  8.121531e-01,  2.665550e-01],
            [ 4.381333e-01, -1.760931e-01, -2.374533e-01, ...,...
    y: array([[[[ 4.410131e-01,  5.410524e-01,  3.447238e-01, ...,
              7.505345e-01,  4.060766e-01,  1.332775e-01],
            [ 2.190667e-01, -1.760931e-01, -2.374533e-01, ...,...
   
   ======================================================================
   FAIL: test_averagepool_2d_same_upper_cpu (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 189 / 3072 (6.15%)
   Max absolute difference: 0.9547153
   Max relative difference: 3.
    x: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
             -0.67406 ],
            [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...
    y: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
             -0.33703 ],
            [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...
   
   ======================================================================
   FAIL: test_averagepool_2d_same_upper_cuda (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 189 / 3072 (6.15%)
   Max absolute difference: 0.9547153
   Max relative difference: 3.
    x: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
             -0.67406 ],
            [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...
    y: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
             -0.33703 ],
            [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...
   
   ======================================================================
   FAIL: test_equal_bcast_cpu (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 1 / 60 (1.67%)
    x: array([[[False, False, False, False, False],
           [False, False, False, False, False],
           [False, False, False, False, False],...
    y: array([[[False, False, False, False, False],
           [False, False, False, False, False],
           [False, False, False, False, False],...
   
   ======================================================================
   FAIL: test_equal_cpu (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 1 / 60 (1.67%)
    x: array([[[False, False, False, False, False],
           [False, False, False, False, False],
           [False, False, False, False,  True],...
    y: array([[[False, False, False, False, False],
           [False, False, False, False, False],
           [False, False, False, False,  True],...
   
   ======================================================================
   FAIL: test_maxpool_2d_same_lower_cpu (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 35 / 3072 (1.14%)
   Max absolute difference: 1.6961312
   Max relative difference: 0.
    x: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
              1.532779e+00,  1.469359e+00,  3.781625e-01],
            [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...
    y: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
              1.532779e+00,  1.469359e+00,  3.781625e-01],
            [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...
   
   ======================================================================
   FAIL: test_maxpool_2d_same_lower_cuda (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 35 / 3072 (1.14%)
   Max absolute difference: 1.6961312
   Max relative difference: 0.
    x: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
              1.532779e+00,  1.469359e+00,  3.781625e-01],
            [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...
    y: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
              1.532779e+00,  1.469359e+00,  3.781625e-01],
            [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...
   
   ======================================================================
   FAIL: test_maxpool_2d_same_upper_cpu (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 37 / 3072 (1.2%)
   Max absolute difference: 1.2028884
   Max relative difference: 0.
    x: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
              0.378163],
            [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...
    y: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
              0.378163],
            [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...
   
   ======================================================================
   FAIL: test_maxpool_2d_same_upper_cuda (__main__.OnnxBackendNodeModelTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
248, in device_test_func
       return test_func(*args, device=device, **kwargs)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
313, in run
       atol=model_test.atol)
     File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line
178, in assert_similar_outputs
       atol=atol)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533,
in assert_allclose
       verbose=verbose, header=header, equal_nan=equal_nan)
     File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846,
in assert_array_compare
       raise AssertionError(msg)
   AssertionError:
   Not equal to tolerance rtol=0.001, atol=1e-07
   
   Mismatched elements: 37 / 3072 (1.2%)
   Max absolute difference: 1.2028884
   Max relative difference: 0.
    x: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
              0.378163],
            [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...
    y: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
              0.378163],
            [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...
   
   ----------------------------------------------------------------------
   Ran 1114 tests in 2.126s
   
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message