<li>Add ResNeSt models (code adapted from <ahref="https://github.com/zhanghang1989/ResNeSt">https://github.com/zhanghang1989/ResNeSt</a>, paper <ahref="https://arxiv.org/abs/2004.08955">https://arxiv.org/abs/2004.08955</a>))</li>
</ul>
<h3id="may-3-2020">May 3, 2020</h3>
<ul>
<li>Pruned EfficientNet B1, B2, and B3 (<ahref="https://arxiv.org/abs/2002.08258">https://arxiv.org/abs/2002.08258</a>) contributed by <ahref="https://github.com/yoniaflalo">Yonathan Aflalo</a></li>
</ul>
<h3id="may-1-2020">May 1, 2020</h3>
<ul>
<li>Merged a number of execellent contributions in the ResNet model family over the past month<ul>
<li>BlurPool2D and resnetblur models initiated by <ahref="https://github.com/VRandme">Chris Ha</a>, I trained resnetblur50 to 79.3.</li>
<li>TResNet models and SpaceToDepth, AntiAliasDownsampleLayer layers by <ahref="https://github.com/mrT23">mrT23</a></li>
<li>ecaresnet (50d, 101d, light) models and two pruned variants using pruning as per (<ahref="https://arxiv.org/abs/2002.08258">https://arxiv.org/abs/2002.08258</a>) by <ahref="https://github.com/yoniaflalo">Yonathan Aflalo</a></li>
</ul>
</li>
<li>200 pretrained models in total now with updated results csv in results folder</li>
</ul>
<h3id="april-5-2020">April 5, 2020</h3>
<ul>
<li>Add some newly trained MobileNet-V2 models trained with latest h-params, rand augment. They compare quite favourably to EfficientNet-Lite<ul>
@ -589,6 +677,21 @@
<li>Add EfficientNet-Lite models w/ weights ported from <ahref="https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite">Tensorflow TPU</a></li>
<li>Add RandAugment trained ResNeXt-50 32x4d weights with 79.8 top-1. Trained by <ahref="https://github.com/andravin">Andrew Lavin</a> (see Training section for hparams)</li>
</ul>
<h3id="april-5-2020_1">April 5, 2020</h3>
<ul>
<li>Add some newly trained MobileNet-V2 models trained with latest h-params, rand augment. They compare quite favourably to EfficientNet-Lite<ul>
<li>3.5M param MobileNet-V2 100 @ 73%</li>
<li>4.5M param MobileNet-V2 110d @ 75%</li>
<li>6.1M param MobileNet-V2 140 @ 76.5%</li>
<li>5.8M param MobileNet-V2 120d @ 77.3%</li>
</ul>
</li>
</ul>
<h3id="march-18-2020_1">March 18, 2020</h3>
<ul>
<li>Add EfficientNet-Lite models w/ weights ported from <ahref="https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite">Tensorflow TPU</a></li>
<li>Add RandAugment trained ResNeXt-50 32x4d weights with 79.8 top-1. Trained by <ahref="https://github.com/andravin">Andrew Lavin</a> (see Training section for hparams)</li>
</ul>
<h3id="feb-29-2020">Feb 29, 2020</h3>
<ul>
<li>New MobileNet-V3 Large weights trained from stratch with this code to 75.77% top-1</li>
<li>More model archs, incl a flexible ByobNet backbone ('Bring-your-own-blocks')<ul>
<li>GPU-Efficient-Networks (<ahref="https://github.com/idstcv/GPU-Efficient-Networks">https://github.com/idstcv/GPU-Efficient-Networks</a>), impl in <code>byobnet.py</code></li>
<li>RepVGG (<ahref="https://github.com/DingXiaoH/RepVGG">https://github.com/DingXiaoH/RepVGG</a>), impl in <code>byobnet.py</code></li>
<li>classic VGG (from torchvision, impl in <code>vgg</code>)</li>
</ul>
</li>
<li>Refinements to normalizer layer arg handling and normalizer+act layer handling in some models</li>
<li>Default AMP mode changed to native PyTorch AMP instead of APEX. Issues not being fixed with APEX. Native works with <code>--channels-last</code> and <code>--torchscript</code> model training, APEX does not.</li>
<li>Fix a few bugs introduced since last pypi release</li>
</ul>
<h3id="feb-8-2021">Feb 8, 2021</h3>
<ul>
<li>Add several ResNet weights with ECA attention. 26t & 50t trained @ 256, test @ 320. 269d train @ 256, fine-tune @320, test @ 352.<ul>
<li>Remove separate tiered (<code>t</code>) vs tiered_narrow (<code>tn</code>) ResNet model defs, all <code>tn</code> changed to <code>t</code> and <code>t</code> models removed (<code>seresnext26t_32x4d</code> only model w/ weights that was removed).</li>
<li>Support model default_cfgs with separate train vs test resolution <code>test_input_size</code> and remove extra <code>_320</code> suffix ResNet model defs that were just for test.</li>
</ul>
<h3id="jan-30-2021">Jan 30, 2021</h3>
<ul>
<li>Add initial "Normalization Free" NF-RegNet-B* and NF-ResNet model definitions based on <ahref="https://arxiv.org/abs/2101.08692">paper</a></li>
</ul>
<h3id="jan-25-2021">Jan 25, 2021</h3>
<ul>
<li>Add ResNetV2 Big Transfer (BiT) models w/ ImageNet-1k and 21k weights from <ahref="https://github.com/google-research/big_transfer">https://github.com/google-research/big_transfer</a></li>
<li>Add official R50+ViT-B/16 hybrid models + weights from <ahref="https://github.com/google-research/vision_transformer">https://github.com/google-research/vision_transformer</a></li>
<li>ImageNet-21k ViT weights are added w/ model defs and representation layer (pre logits) support<ul>
<li>NOTE: ImageNet-21k classifier heads were zero'd in original weights, they are only useful for transfer learning</li>
</ul>
</li>
<li>Add model defs and weights for DeiT Vision Transformer models from <ahref="https://github.com/facebookresearch/deit">https://github.com/facebookresearch/deit</a></li>
<li>Refactor dataset classes into ImageDataset/IterableImageDataset + dataset specific parser classes</li>
<li>Add Tensorflow-Datasets (TFDS) wrapper to allow use of TFDS image classification sets with train script<ul>
<li>Add improved .tar dataset parser that reads images from .tar, folder of .tar files, or .tar within .tar<ul>
<li>Run validation on full ImageNet-21k directly from tar w/ BiT model: <code>validate.py /data/fall11_whole.tar --model resnetv2_50x1_bitm_in21k --amp</code></li>
</ul>
</li>
<li>Models in this update should be stable w/ possible exception of ViT/BiT, possibility of some regressions with train/val scripts and dataset handling</li>
</ul>
<h3id="jan-3-2021">Jan 3, 2021</h3>
<ul>
<li>Add SE-ResNet-152D weights<ul>
<li>256x256 val, 0.94 crop top-1 - 83.75</li>
<li>320x320 val, 1.0 crop - 84.36</li>
</ul>
</li>
<li>Update results files</li>
</ul>
<h3id="dec-18-2020">Dec 18, 2020</h3>
<ul>
<li>Add ResNet-101D, ResNet-152D, and ResNet-200D weights trained @ 256x256<ul>
<li>trained from scratch using the included training script</li>
</ol>
<p>The validation results for the pretrained weights can be found <ahref="../results/">here</a></p>
<h2id="big-transfer-resnetv2-bit-resnetv2py">Big Transfer ResNetV2 (BiT) [<ahref="https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/resnetv2.py">resnetv2.py</a>]</h2>
<ul>
<li>Paper: <code>Big Transfer (BiT): General Visual Representation Learning</code> - <ahref="https://arxiv.org/abs/1912.11370">https://arxiv.org/abs/1912.11370</a></li>
<li>Paper: <code>CSPNet: A New Backbone that can Enhance Learning Capability of CNN</code> - <ahref="https://arxiv.org/abs/1911.11929">https://arxiv.org/abs/1911.11929</a></li>
<li>Paper: <code>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</code> - <ahref="https://arxiv.org/abs/2010.11929">https://arxiv.org/abs/2010.11929</a></li>
<p>The variety of training args is large and not all combinations of options (or even options) have been fully tested. For the training dataset folder, specify the folder to the base that contains a <code>train</code> and <code>validation</code> folder.</p>
<p>To train an SE-ResNet34 on ImageNet, locally distributed, 4 GPUs, one process per GPU w/ cosine schedule, random-erasing prob of 50% and per-pixel random value:</p>
<p>NOTE: It is recommended to use PyTorch 1.7+ w/ PyTorch native AMP and DDP instead of APEX AMP. <code>--amp</code> defaults to native AMP as of timm ver 0.4.3. <code>--apex-amp</code> will force use of APEX components if they are installed.</p>
<p>Validation and inference scripts are similar in usage. One outputs metrics on a validation set and the other outputs topk class ids in a csv. Specify the folder containing validation images, not the base as in training script. </p>
<p>To validate with the model's pretrained weights (if they exist):</p>
Ross Wightman
3 years ago