ab16a358bb
Add log and continue handler for WDS errors, fix args.num_gpu for validation script fallback
3 years ago
7eeaf521a0
use gopen in wds to open info file in case it's at a url/gs location
3 years ago
229ac6b8d8
Fix alternate label handling in WDS parser to skip invalid alt labels
3 years ago
a444d4b891
Add alternative label support to WDS for imagenet22k/12k split, add 21k/22k/12k indices filters to results/
3 years ago
da2796ae82
Add webdataset (WDS) support, update TFDS to make some naming in parsers more similar. Fix workers=0 compatibility. Add ImageNet22k/12k synset defs.
3 years ago
c639a86c67
Change TFDS default to full re-shuffle (init) each epoch (for now)
3 years ago
ff0f709c20
Testing TFDS shuffle across epochs
3 years ago
cad170e494
Merge remote-tracking branch 'origin/norm_norm_norm' into bits_and_tpu
3 years ago
55adfbeb8d
Add commented code to increase open file limit via Python (for TFDS dataset building)
3 years ago
809c7bb1ec
Merge remote-tracking branch 'origin/master' into bits_and_tpu
3 years ago
cfa414cad2
Matching two bits_and_tpu changes for TFDs wrapper
...
* change 'samples' -> 'examples' for tfds wrapper to match tfds naming
* add class_to_idx for image classification datasets in tfds wrapper
3 years ago
4f338556d8
Fixes and improvements for metrics, tfds parser, loader / transform handling
...
* add back ability to create transform with loader
* change 'samples' -> 'examples' for tfds wrapper to match tfds naming
* add support for specifying feature names for input and target in tfds wrapper
* add class_to_idx for image classification datasets in tfds wrapper
* add accumulate_type to avg meters and metrics to allow float32 or float64 accumulation control with lower prec data
* minor cleanup, log output rate prev and avg
3 years ago
406c486ba2
Merge remote-tracking branch 'origin/more_datasets' into bits_and_tpu
3 years ago
9ec3210c2d
More TFDS parser cleanup, support improved TFDS even_split impl (on tfds-nightly only currently).
3 years ago
ba65dfe2c6
Dataset work
...
* support some torchvision datasets
* improvements to TFDS wrapper for subsplit handling (fix #942 ), shuffle seed
* add class-map support to train (fix #957 )
3 years ago
25d52ea71d
Merge remote-tracking branch 'origin/fixes_bce_regnet' into bits_and_tpu
3 years ago
b57a03bd0d
Merge branch 'master' into bits_and_tpu
4 years ago
91ab0b6ce5
Add proper TrainState checkpoint save/load. Some reorg/refactoring and other cleanup. More to go...
4 years ago
aa92d7b1c5
Major timm.bits update. Updater and DeviceEnv now dataclasses, after_step closure used, metrics base impl w/ distributed reduce, many tweaks/fixes.
4 years ago
d53e91218e
Fix tf.data options setting for newer TF versions
4 years ago
938716c753
Fix import issue, use devenv for dist info in parser_tfds
4 years ago
f42f1df26c
Improve evenness of per-worker split for validation set with TFDS
4 years ago
2db2d87ff7
Add epoch-repeats arg to multiply the number of dataset passes per epoch. Currently for iterable datasets (read TFDS wrapper) only.
4 years ago
22748f1a2d
Convert samples/targets in ParserImageInTar to numpy arrays, slightly less mem usage for massive datasets. Add a few more se/eca model defs to resnet.py
4 years ago
5d4c3d0af3
Add enhanced ParserImageInTar that can read images from tars within tars, folders with multiple tars, etc. Additional comment cleanup.
4 years ago
855d6cc217
More dataset work including factories and a tensorflow datasets (TFDS) wrapper
...
* Add parser/dataset factory methods for more flexible dataset & parser creation
* Add dataset parser that wraps TFDS image classification datasets
* Tweak num_classes handling bug for 21k models
* Add initial deit models so they can be benchmarked in next csv results runs
4 years ago
59ec7e6a53
Merge branch 'master' into imagenet21k_datasets_more
4 years ago
e35e9760a6
More work on dataset / parser split and imagenet21k (tar) support
4 years ago
de6046e213
Initial commit for dataset / parser reorg to support additional datasets / types
4 years ago
Ross Wightman