Severe sensitivity to Horizontal Flip

Summary

A Horizontal Flip test was performed on poolformer_s36, in which a 5% failure rate was observed. In at least one case, the model's prediction changed 0.50.

Test Information

This test measures the robustness of the model to Horizontal Flip transformations. It does this by taking a sample input, flipping the image horizontally, and measuring the behavior of the model on the transformed input.

Why is this important?

Production inputs can have unusual variations amongst many different dimensions, ranging from lighting changes to sensor errors to compression artifacts. It is important that the models are robust to the introduction of such variations.

Model Information

• Model name: poolformer_s36
• Model package URL: pkg:huggingface/sail/poolformer_s36@1d740b215d07f480fa51efa16df83a63d5f6acf2
• Macro F1 score on reference / evaluation: 0.90 / 0.80
• Multiclass accuracy on reference / evaluation: 0.90 / 0.80
• Multiclass AUC on reference / evaluation: 1.00 / 1.00
• Macro Precision on reference / evaluation: 0.92 / 0.82
• Macro Recall on reference / evaluation: 0.90 / 0.80

This report was automatically generated by the scanning engine rime-0.21.0rc4.post195+git.2a88076b.d on 2023-01-07 04:40.