12 Commits (vit_relpos_refactor)

Author	SHA1	Message	Date
Alexander Soare	b25ff96768	wip - pre-rebase	4 years ago
Alexander Soare	e051dce354	Make all models FX traceable	4 years ago
Alexander Soare	0149ec30d7	wip - attempting to rebase	4 years ago
Alexander Soare	bc3d4eb403	wip -rebase	4 years ago
Ross Wightman	02daf2ab94	Add option to include relative pos embedding in the attention scaling as per references. See discussion #912	4 years ago
Ross Wightman	e2b8d44ff0	Halo, bottleneck attn, lambda layer additions and cleanup along w/ experimental model defs ... * align interfaces of halo, bottleneck attn and lambda layer * add qk_ratio to all of above, control q/k dim relative to output dim * add experimental haloregnetz, and trionet (lambda + halo + bottle) models	4 years ago
Ross Wightman	007bc39323	Some halo and bottleneck attn code cleanup, add halonet50ts weights, use optimal crop ratios	4 years ago
Ross Wightman	b81e79aae9	Fix bottleneck attn transpose typo, hopefully these train better now..	4 years ago
Ross Wightman	5bd04714e4	Cleanup weight init for byob/byoanet and related	4 years ago
Ross Wightman	8642401e88	Swap botnet 26/50 weights/models after realizing a mistake in arch def, now figuring out why they were so low...	4 years ago
Ross Wightman	0721559511	Improved (hopefully) init for SA/SA-like layers used in ByoaNets	4 years ago
Ross Wightman	ce62f96d4d	ByoaNet with bottleneck transformer, lambda resnet, and halo net experiments	4 years ago