81ca323751
Davit update formatting and fix grad checkpointing ( #7 )
...
fixed head to gap->norm->fc as per convnext, along with option for norm->gap->fc
failed tests due to clip convnext models, davit tests passed
2 years ago
927f031293
Major module / path restructure, timm.models.layers -> timm.layers, add _ prefix to all non model modules in timm.models
2 years ago
0862e6ebae
Fix correctness of some group matching regex (no impact on result), some formatting, missed forward_head for resnet
3 years ago
94bcdebd73
Add latest weights trained on TPU-v3 VM instances
3 years ago
372ad5fa0d
Significant model refactor and additions:
...
* All models updated with revised foward_features / forward_head interface
* Vision transformer and MLP based models consistently output sequence from forward_features (pooling or token selection considered part of 'head')
* WIP param grouping interface to allow consistent grouping of parameters for layer-wise decay across all model types
* Add gradient checkpointing support to a significant % of models, especially popular architectures
* Formatting and interface consistency improvements across models
* layer-wise LR decay impl part of optimizer factory w/ scale support in scheduler
* Poolformer and Volo architectures added
3 years ago
abc9ba2544
Transitioning default_cfg -> pretrained_cfg. Improving handling of pretrained_cfg source (HF-Hub, files, timm config, etc). Checkpoint handling tweaks.
3 years ago
ab49d275de
Significant norm update
...
* ConvBnAct layer renamed -> ConvNormAct and ConvNormActAa for anti-aliased
* Significant update to EfficientNet and MobileNetV3 arch to support NormAct layers and grouped conv (as alternative to depthwise)
* Update RegNet to add Z variant
* Add Pre variant of XceptionAligned that works with NormAct layers
* EvoNorm matches bits_and_tpu branch for merge
3 years ago
78912b6375
Updated EvoNorm implementations with some experimentation. Add FilterResponseNorm. Updated RegnetZ and ResNetV2 model defs for trials.
3 years ago
734b2244fe
Add RegNetZ-D8 (83.5 @ 256, 84 @ 320) and RegNetZ-E8 (84.5 @ 256, 85 @ 320) weights. Update names of existing RegZ models to include group size.
3 years ago
0fe4fd3f1f
add d8 and e8 regnetz models with group size 8
3 years ago
c02334d9fa
Add weights for regnetz_d and haloregnetz_c, update regnetz_c weights. Add commented PyTorch XLA code for halo attention
3 years ago
b544ad4d3f
regnetz model default cfg tweaks
3 years ago
e2b8d44ff0
Halo, bottleneck attn, lambda layer additions and cleanup along w/ experimental model defs
...
* align interfaces of halo, bottleneck attn and lambda layer
* add qk_ratio to all of above, control q/k dim relative to output dim
* add experimental haloregnetz, and trionet (lambda + halo + bottle) models
3 years ago
0ca687f224
Make 'regnetz' model experiments closer to actual RegNetZ, bottleneck expansion, expand from in_chs, no shortcut on stride 2, tweak model sizes
3 years ago
6478bcd02c
Fix regnetz_d conv layer name, use inception mean/std
3 years ago
8e11da0ce3
Add experimental RegNetZ(ish) models for training / perf trials.
3 years ago
4027412757
Add resnet33ts weights, update resnext26ts baseline weights
3 years ago
5bd04714e4
Cleanup weight init for byob/byoanet and related
3 years ago
484e61648d
Adding the attn series weights, tweaking model names, comments...
3 years ago
8449ba210c
Improve performance of HaloAttn, change default dim calc. Some cleanup / fixes for byoanet. Rename resnet26ts to tfs to distinguish (extra fc).
3 years ago
a8b65695f1
Add resnet26ts and resnext26ts models for non-attn baselines
3 years ago
a5a542f17d
Fix typo
3 years ago
925e102982
Update attention / self-attn based models from a series of experiments:
...
* remove dud attention, involution + my swin attention adaptation don't seem worth keeping
* add or update several new 26/50 layer ResNe(X)t variants that were used in experiments
* remove models associated with dead-end or uninteresting experiment results
* weights coming soon...
3 years ago
392368e210
Add efficientnetv2_rw_t defs w/ weights, and gc variant, as well as gcresnet26ts for experiments. Version 0.4.13
4 years ago
d17b374f0f
Minimum input_size needed to be higher
4 years ago
b3b90d944d
Add min_input_size to bat_resnext to prevent test breakage.
4 years ago
10d8fa4620
Add gc and bat attention resnext26ts variants to byob for test.
4 years ago
307a935b79
Add non-local and BAT attention. Merge attn and self-attn factories into one. Add attention references to README. Add mlp 'mode' to ECA.
4 years ago
9611458e19
Throw in some FBNetV3 code I had lying around, some refactoring of SE reduction channel calcs for all EffNet archs.
4 years ago
02f9d4bc34
Add weights for resnet51q model, add 61q def.
4 years ago
f615474be3
Fix broken test, repvgg block doesn't have attn_last attr.
4 years ago
742c2d5247
Add Gather-Excite and Global Context attn modules. Refactor existing SE-like attn for consistency and refactor byob/byoanet for less redundancy.
4 years ago
9a3ae97311
Another set of byoanet models w/ ECA channel + SA + groups
4 years ago
b3d7580df1
Update ByoaNet comments. Fix first Steam feat chs for ByobNet.
4 years ago
7c97e66f7c
Remove commented code, add more consistent seed fn
4 years ago
ce62f96d4d
ByoaNet with bottleneck transformer, lambda resnet, and halo net experiments
4 years ago
6853b07bbd
Improve RegVGG block identity/vs non for clariy and fix attn usage. Add comments.
4 years ago
dc85e5a237
Add ByobNet w/ GPU-EfficientNets and RepVGG. Also add classic vgg models.
4 years ago
Fredo Guan