From 1ffd2d06949670c05d6983c395b01d5e36d70e45 Mon Sep 17 00:00:00 2001 From: Ross Wightman Date: Sun, 2 Feb 2020 19:35:44 -0800 Subject: [PATCH 1/4] Add results/README.md --- results/README.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 results/README.md diff --git a/results/README.md b/results/README.md new file mode 100644 index 00000000..40bb8682 --- /dev/null +++ b/results/README.md @@ -0,0 +1,39 @@ +# Validation Results + +This folder contains validation results for the models in this collection having pretrained weights. Since the focus for this repository is currently ImageNet-1k classification, all of the results are based on datasets compatible with ImageNet-1k classes. + +## Datasets + +There are currently results for the ImageNet validation set and 3 additional test sets. + +### ImageNet Validation - [`results-imagenet.csv`](results/results-imagenet.csv) + +* Source: http://image-net.org/challenges/LSVRC/2012/index +* Paper: "ImageNet Large Scale Visual Recognition Challenge" - https://arxiv.org/abs/1409.0575 + +The standard 50,000 image ImageNet-1k validation set. Model selection during training utilizes this validation set, so it is not a true test set. + +### ImageNetV2 Matched Frequency - [`results-imagenetv2-matched-frequency.csv`](results/results-imagenetv2-matched-frequency.csv) + +* Source: https://github.com/modestyachts/ImageNetV2 +* Paper: "Do ImageNet Classifiers Generalize to ImageNet?" - https://arxiv.org/abs/1902.10811 + +An ImageNet test set of 10,000 images sampled from new images roughly 10 years after the original. Care was taken to replicate the original ImageNet curation/sampling process. + +### ImageNet-Sketch - [`results-sketch.csv`](results/results-imagenet-sketch.csv) + +* Source: https://github.com/HaohanWang/ImageNet-Sketch +* Paper: "Learning Robust Global Representations by Penalizing Local Predictive Power" - https://arxiv.org/abs/1905.13549 + +50,000 non photographic (or photos of such) images (sketches, doodles, mostly monochromatic) covering all 1000 ImageNet classes. + +### ImageNet-Adversarial - [`results-imagenet-a.csv`](results/results-imagenet-a.csv) + +* Source: https://github.com/hendrycks/natural-adv-examples +* Paper: "Natural Adversarial Examples" - https://arxiv.org/abs/1907.07174 + +A collection of 7500 images covering 200 of the 1000 ImageNet classes. Images are naturally occuring adversarial examples that confuse typical ImageNet classifiers. This is a challenging dataset, your average ResNet-50 will score 0% top-1. + +## TODO +* Add rank difference, and top-1/top-5 difference from ImageNet-1k validation for the 3 additional test sets +* Explore adding a reduced version of ImageNet-C (Corruptions) and ImageNet-P (Perturbations) from https://github.com/hendrycks/robustness. The originals are huge and image size specific. From 82c0a2ff1279de472b704330f046acfb4b6bbe9f Mon Sep 17 00:00:00 2001 From: Ross Wightman Date: Sun, 2 Feb 2020 19:37:18 -0800 Subject: [PATCH 2/4] Update README.md Fix relative paths (I think) --- results/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/results/README.md b/results/README.md index 40bb8682..6bb160d7 100644 --- a/results/README.md +++ b/results/README.md @@ -6,28 +6,28 @@ This folder contains validation results for the models in this collection having There are currently results for the ImageNet validation set and 3 additional test sets. -### ImageNet Validation - [`results-imagenet.csv`](results/results-imagenet.csv) +### ImageNet Validation - [`results-imagenet.csv`](results-imagenet.csv) * Source: http://image-net.org/challenges/LSVRC/2012/index * Paper: "ImageNet Large Scale Visual Recognition Challenge" - https://arxiv.org/abs/1409.0575 The standard 50,000 image ImageNet-1k validation set. Model selection during training utilizes this validation set, so it is not a true test set. -### ImageNetV2 Matched Frequency - [`results-imagenetv2-matched-frequency.csv`](results/results-imagenetv2-matched-frequency.csv) +### ImageNetV2 Matched Frequency - [`results-imagenetv2-matched-frequency.csv`](results-imagenetv2-matched-frequency.csv) * Source: https://github.com/modestyachts/ImageNetV2 * Paper: "Do ImageNet Classifiers Generalize to ImageNet?" - https://arxiv.org/abs/1902.10811 An ImageNet test set of 10,000 images sampled from new images roughly 10 years after the original. Care was taken to replicate the original ImageNet curation/sampling process. -### ImageNet-Sketch - [`results-sketch.csv`](results/results-imagenet-sketch.csv) +### ImageNet-Sketch - [`results-sketch.csv`](results-imagenet-sketch.csv) * Source: https://github.com/HaohanWang/ImageNet-Sketch * Paper: "Learning Robust Global Representations by Penalizing Local Predictive Power" - https://arxiv.org/abs/1905.13549 50,000 non photographic (or photos of such) images (sketches, doodles, mostly monochromatic) covering all 1000 ImageNet classes. -### ImageNet-Adversarial - [`results-imagenet-a.csv`](results/results-imagenet-a.csv) +### ImageNet-Adversarial - [`results-imagenet-a.csv`](results-imagenet-a.csv) * Source: https://github.com/hendrycks/natural-adv-examples * Paper: "Natural Adversarial Examples" - https://arxiv.org/abs/1907.07174 From 820b73d2800a1e6d0c4836ec0d45d39e07bddac3 Mon Sep 17 00:00:00 2001 From: Ross Wightman Date: Sun, 2 Feb 2020 19:38:10 -0800 Subject: [PATCH 3/4] Update README.md --- results/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/results/README.md b/results/README.md index 6bb160d7..ee1f941d 100644 --- a/results/README.md +++ b/results/README.md @@ -20,7 +20,7 @@ The standard 50,000 image ImageNet-1k validation set. Model selection during tra An ImageNet test set of 10,000 images sampled from new images roughly 10 years after the original. Care was taken to replicate the original ImageNet curation/sampling process. -### ImageNet-Sketch - [`results-sketch.csv`](results-imagenet-sketch.csv) +### ImageNet-Sketch - [`results-sketch.csv`](results-sketch.csv) * Source: https://github.com/HaohanWang/ImageNet-Sketch * Paper: "Learning Robust Global Representations by Penalizing Local Predictive Power" - https://arxiv.org/abs/1905.13549 From 5c85389e268d01a95a306e7a8d39f33ee3486976 Mon Sep 17 00:00:00 2001 From: Ross Wightman Date: Sun, 2 Feb 2020 20:01:26 -0800 Subject: [PATCH 4/4] Update README.md --- results/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/results/README.md b/results/README.md index ee1f941d..1e9a1795 100644 --- a/results/README.md +++ b/results/README.md @@ -11,7 +11,7 @@ There are currently results for the ImageNet validation set and 3 additional tes * Source: http://image-net.org/challenges/LSVRC/2012/index * Paper: "ImageNet Large Scale Visual Recognition Challenge" - https://arxiv.org/abs/1409.0575 -The standard 50,000 image ImageNet-1k validation set. Model selection during training utilizes this validation set, so it is not a true test set. +The standard 50,000 image ImageNet-1k validation set. Model selection during training utilizes this validation set, so it is not a true test set. Question: Does anyone have the official ImageNet-1k test set classification labels now that challenges are done? ### ImageNetV2 Matched Frequency - [`results-imagenetv2-matched-frequency.csv`](results-imagenetv2-matched-frequency.csv) @@ -32,7 +32,7 @@ An ImageNet test set of 10,000 images sampled from new images roughly 10 years a * Source: https://github.com/hendrycks/natural-adv-examples * Paper: "Natural Adversarial Examples" - https://arxiv.org/abs/1907.07174 -A collection of 7500 images covering 200 of the 1000 ImageNet classes. Images are naturally occuring adversarial examples that confuse typical ImageNet classifiers. This is a challenging dataset, your average ResNet-50 will score 0% top-1. +A collection of 7500 images covering 200 of the 1000 ImageNet classes. Images are naturally occuring adversarial examples that confuse typical ImageNet classifiers. This is a challenging dataset, your typical ResNet-50 will score 0% top-1. ## TODO * Add rank difference, and top-1/top-5 difference from ImageNet-1k validation for the 3 additional test sets