tvm-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-tvm] masahi edited a comment on issue #5061: [Torch, QNN] Remove FP32 piggy back and use QNN add/mul/concatenate
Date Fri, 13 Mar 2020 20:42:47 GMT
masahi edited a comment on issue #5061: [Torch, QNN] Remove FP32 piggy back and use QNN add/mul/concatenate
URL: https://github.com/apache/incubator-tvm/pull/5061#issuecomment-598902899
 
 
   @anijain2305 @jwfromm @jjohnson-arm 
   
   Here is the current result on mobilenet v2, using QNN `add` and post training calibration
(**which is wrong**).
   
   
   ```
   Model name: mobilenet_v2, per channel quantization
   PyTorch accuracy: Top1 = 67.87, Top5 = 88.15
   TVM accuracy: Top1 = 62.47, Top5 = 84.67
   PyTorch top5 label: [101 386  51 385  69]
   TVM top5 label: [101 386  51 385 340]
   PyTorch top5 raw output: [18.233843 16.314491 15.674707 13.115572 12.795679]
   TVM top5 raw output: [27.510712 26.231144 21.752655 20.153194 17.274168]
   max abs diff: 9.916653
   mean abs_diff: 2.0649028
   50 in 1000 raw outputs correct.
   ```
   
   We lost 5 point accuracy compared to Torch.
   
   And here is **without** post training calibration, also using QNN `add`. Now the top1 accuracy
is much better and almost the same as Torch. Moreover, the raw output of the network, 1000
floating point values, are much closer to Torch. The former has only 50 out of 1000 outputs
identical, while the latter, correct one has 376/1000.
   
   ```
   Model name: mobilenet_v2, per channel quantization
   PyTorch accuracy: Top1 = 71.32, Top5 = 89.86
   TVM accuracy: Top1 = 71.27, Top5 = 89.86
   PyTorch top5 label: [101 386 385  51 340]
   TVM top5 label: [101 386 385  51 340]
   PyTorch top5 raw output: [20.168097 18.80845  17.222195 13.59647   9.290921]
   TVM top5 raw output: [19.941488 18.581842 16.995586 13.823077  9.064313]
   max abs diff: 0.9064312
   mean abs_diff: 0.17562106
   376 in 1000 raw outputs correct.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message