Accuracy
* Official reported data is put in parentheses
| Model |
Top-1 |
Top-5 |
| alexnet |
55.9%(54.9%) |
78.7%(78.0%) |
| densenet121 |
74.9%(75.0%) |
92.2%(92.3%) |
| densenet169 |
76.1%(76.2%) |
93.1%(93.2%) |
| densenet201 |
77.3%(77.3%) |
93.7%(93.6%) |
| inception_v3 |
79.0%(78.8%) |
94.4%(94.4%) |
| mobilenet_v2_1_0 |
72.0%(72.0%) |
90.6%(90.6%) |
| mobilenet0_25 |
52.9%(52.9%) |
77.0%(76.9%) |
| mobilenet0_5 |
65.2%(65.2%) |
86.4%(86.3%) |
| mobilenet1_0 |
73.3%(73.3%) |
91.3%(91.3%) |
| resnet101_v1 |
78.4%(78.3%) |
94.0%(94.0%) |
| resnet101_v2 |
78.6%(78.5%) |
94.2%(94.2%) |
| resnet152_v1 |
79.2%(79.2%) |
94.7%(94.6%) |
| resnet152_v2 |
79.2%(79.2%) |
94.3%(94.3%) |
| resnet50_v1 |
77.4%(77.4%) |
93.6%(93.6%) |
| resnet50_v2 |
77.1%(77.1%) |
93.4%(93.4%) |
| resnext50_32x4d |
79.3%(79.3%) |
94.5%(94.5%) |
| squeezenet1_0 |
57.2%(56.1%) |
80.0%(79.1%) |
| vgg16 |
73.2%(73.2%) |
91.3%(91.3%) |
| vgg19 |
74.1%(74.1%) |
91.8%(91.4%) |
| xception |
80.0%(79.6%) |
94.8%(94.8%) |
Neuron Coverage
| Model |
Layers |
Neurons |
t=0.0 |
t=0.1 |
t=0.2 |
t=0.3 |
t=0.4 |
t=0.5 |
t=0.6 |
t=0.7 |
t=0.8 |
t=0.9 |
| alexnet |
18 |
20200 |
99.9% |
99.4% |
97.4% |
95.2% |
93.4% |
92.1% |
91.1% |
90.0% |
87.0% |
81.3% |
| densenet121 |
368 |
96872 |
98.3% |
91.9% |
81.7% |
71.4% |
63.5% |
58.7% |
55.5% |
34.5% |
7.6% |
1.7% |
| densenet169 |
512 |
176360 |
97.2% |
88.8% |
80.6% |
71.1% |
62.5% |
57.2% |
54.1% |
35.6% |
7.5% |
1.2% |
| densenet201 |
608 |
250344 |
96.5% |
86.8% |
77.3% |
67.8% |
59.8% |
55.6% |
52.4% |
34.6% |
5.8% |
0.9% |
| inception_v3 |
297 |
63144 |
100.0% |
98.1% |
89.7% |
81.3% |
72.1% |
64.3% |
57.9% |
38.9% |
18.2% |
6.3% |
| mobilenet_v2_1_0 |
106 |
36392 |
100.0% |
100.0% |
100.0% |
100.0% |
100.0% |
100.0% |
97.9% |
73.4% |
29.2% |
8.0% |
| mobilenet0_25 |
83 |
9464 |
100.0% |
99.5% |
95.7% |
90.0% |
83.9% |
77.5% |
68.5% |
50.7% |
28.5% |
15.2% |
| mobilenet0_5 |
83 |
17928 |
99.9% |
99.2% |
94.0% |
87.7% |
81.4% |
75.2% |
66.0% |
46.1% |
22.6% |
10.0% |
| mobilenet1_0 |
83 |
34856 |
100.0% |
98.4% |
91.3% |
84.8% |
78.2% |
72.3% |
63.2% |
45.6% |
21.6% |
9.6% |
| resnet101_v1 |
278 |
124776 |
100.0% |
99.8% |
95.6% |
91.7% |
89.3% |
87.9% |
79.4% |
35.8% |
5.4% |
2.2% |
| resnet101_v2 |
211 |
106795 |
100.0% |
99.2% |
99.1% |
98.9% |
98.7% |
98.2% |
90.5% |
43.0% |
6.7% |
2.2% |
| resnet152_v1 |
414 |
178536 |
100.0% |
99.2% |
93.3% |
89.6% |
87.8% |
86.9% |
84.2% |
53.1% |
7.5% |
1.8% |
| resnet152_v2 |
313 |
152875 |
100.0% |
99.1% |
99.1% |
99.0% |
97.6% |
96.8% |
91.9% |
43.9% |
6.2% |
1.5% |
| resnet50_v1 |
142 |
63848 |
100.0% |
99.6% |
96.7% |
93.9% |
91.1% |
89.0% |
81.2% |
44.3% |
12.6% |
5.0% |
| resnet50_v2 |
109 |
54571 |
99.9% |
98.6% |
98.3% |
98.0% |
97.4% |
95.7% |
82.5% |
48.1% |
9.0% |
4.3% |
| resnext50_32x4d |
142 |
86504 |
100.0% |
97.2% |
91.8% |
87.9% |
84.9% |
81.6% |
77.9% |
49.0% |
14.5% |
4.7% |
| squeezenet1_0 |
56 |
9816 |
100.0% |
97.3% |
88.0% |
75.1% |
60.1% |
54.5% |
49.4% |
29.4% |
13.8% |
10.5% |
| vgg16 |
36 |
27304 |
100.0% |
97.3% |
89.4% |
84.4% |
81.4% |
79.9% |
79.3% |
73.6% |
64.8% |
63.5% |
| vgg19 |
42 |
29864 |
100.0% |
96.9% |
88.0% |
82.7% |
79.3% |
77.5% |
76.5% |
66.9% |
58.8% |
58.0% |
| xception |
331 |
234520 |
100.0% |
93.3% |
87.4% |
83.7% |
81.4% |
80.6% |
76.2% |
49.7% |
8.5% |
2.5% |
Robustness
| Model |
FGSM |
BIM |
DeepFool* |
| Success Rate |
Avg Time |
Avg Linf Dist |
Success Rate |
Avg Time |
Avg Linf Dist |
Success Rate |
Avg Time |
Avg MSE |
| alexnet |
100.0% |
0.13s |
0.00244221 |
100.0% |
2.11s |
0.00141288 |
100.0% |
0.30s |
0.00000157 |
| densenet121 |
100.0% |
4.85s |
0.10966790 |
100.0% |
22.69s |
0.06918367 |
NA |
NA |
NA |
| densenet169 |
100.0% |
7.51s |
0.12484919 |
100.0% |
31.82s |
0.07060633 |
NA |
NA |
NA |
| densenet201 |
100.0% |
8.62s |
0.11938908 |
100.0% |
38.71s |
0.07444767 |
NA |
NA |
NA |
| inception_v3 |
100.0% |
6.48s |
0.18367064 |
100.0% |
19.46s |
0.04420286 |
NA |
NA |
NA |
| mobilenet_v2_1_0 |
99.9% |
1.71s |
0.09063636 |
100.0% |
10.23s |
0.05450443 |
NA |
NA |
NA |
| mobilenet0_25 |
99.6% |
0.60s |
0.04284756 |
100.0% |
4.98s |
0.02777374 |
NA |
NA |
NA |
| mobilenet0_5 |
100.0% |
0.79s |
0.07233644 |
100.0% |
5.52s |
0.04514066 |
NA |
NA |
NA |
| mobilenet1_0 |
100.0% |
1.24s |
0.12316780 |
100.0% |
6.26s |
0.08263142 |
NA |
NA |
NA |
| resnet101_v1 |
100.0% |
4.52s |
0.10904009 |
100.0% |
23.81s |
0.07088844 |
NA |
NA |
NA |
| resnet101_v2 |
100.0% |
5.60s |
0.14149920 |
100.0% |
23.95s |
0.09782901 |
NA |
NA |
NA |
| resnet152_v1 |
100.0% |
6.72s |
0.11078327 |
100.0% |
35.35s |
0.05255808 |
NA |
NA |
NA |
| resnet152_v2 |
100.0% |
7.80s |
0.13422088 |
100.0% |
35.42s |
0.09334203 |
NA |
NA |
NA |
| resnet50_v1 |
100.0% |
1.87s |
0.08280256 |
100.0% |
12.56s |
0.02140497 |
NA |
NA |
NA |
| resnet50_v2 |
100.0% |
2.52s |
0.12011731 |
100.0% |
12.51s |
0.08649258 |
NA |
NA |
NA |
| resnext50_32x4d |
100.0% |
3.85s |
0.15879153 |
100.0% |
18.45s |
0.11347104 |
NA |
NA |
NA |
| squeezenet1_0 |
100.0% |
0.17s |
0.00171386 |
100.0% |
3.36s |
0.00083149 |
NA |
NA |
NA |
| vgg16 |
100.0% |
0.63s |
0.00682080 |
100.0% |
11.09s |
0.00153889 |
98.8% |
2.80s |
0.00000057 |
| vgg19 |
100.0% |
0.88s |
0.01106281 |
100.0% |
13.62s |
0.00259408 |
98.7% |
3.34s |
0.00000063 |
| xception |
100.0% |
8.89s |
0.14150371 |
100.0% |
44.67s |
0.15265589 |
NA |
NA |
NA |
* We cannot achieve a reasonable attack success rate when using Foolbox DeepFool on most of the MXNet model. We are discussing this issue with Foolbox.
file