11 Commits (7cedc8d4743f2b2bbf835fc387c917461fa4911a)

Author	SHA1	Message	Date
Ross Wightman	a16a753852	Add lamb/lars to optim init imports, remove stray comment	3 years ago
Ross Wightman	8a9eca5157	A few optimizer comments, dead import, missing import	3 years ago
Ross Wightman	ac469b50da	Optimizer improvements, additions, cleanup ... * Add MADGRAD code * Fix Lamb (non-fused variant) to work w/ PyTorch XLA * Tweak optimizer factory args (lr/learning_rate and opt/optimizer_name), may break compat * Use newer fn signatures for all add,addcdiv, addcmul in optimizers * Use upcoming PyTorch native Nadam if it's available * Cleanup lookahead opt * Add optimizer tests * Remove novograd.py impl as it was messy, keep nvnovograd * Make AdamP/SGDP work in channels_last layout * Add rectified adablief mode (radabelief) * Support a few more PyTorch optim, adamax, adagrad	3 years ago
juntang	addfc7c1ac	adabelief	3 years ago
Ross Wightman	288682796f	Update benchmark script to add precision arg. Fix some downstream (DeiT) compat issues with latest changes. Bump version to 0.4.7	3 years ago
Ross Wightman	0e16d4e9fb	Add benchmark.py script, and update optimizer factory to be more friendly to use outside of argparse interface.	3 years ago
Ross Wightman	80078c47bb	Add Adafactor and Adahessian optimizers, cleanup optimizer arg passing, add gradient clipping support.	4 years ago
Sangdoo Yun	e93e571f7a	Add `adamp` and 'sgdp' optimizers. ... Update requirements.txt Update optim_factory.py Add `adamp` optimizer Update __init__.py copy files of adamp & sgdp Create adamp.py Update __init__.py Create sgdp.py Update optim_factory.py Update optim_factory.py Update requirements.txt Update adamp.py Update sgdp.py Update sgdp.py Update adamp.py	4 years ago
Ross Wightman	64966f61f7	Add Nvidia's NovogGrad impl from Jasper (cleaner/faster than current) and Apex Fused optimizers	5 years ago
Ross Wightman	fac58f609a	Add RAdam, NovoGrad, Lookahead, and AdamW optimizers, a few ResNet tweaks and scheduler factory tweak. ... * Add some of the trendy new optimizers. Decent results but not clearly better than the standards. * Can create a None scheduler for constant LR * ResNet defaults to zero_init of last BN in residual * add resnet50d config	5 years ago
Ross Wightman	aa4354f466	Big re-org, working towards making pip/module as 'timm'	5 years ago