|
|
|
@ -2,41 +2,80 @@
|
|
|
|
|
|
|
|
|
|
![AppIcon](images/appicon180.png)
|
|
|
|
|
|
|
|
|
|
![Image](images/ss0_1280.png)
|
|
|
|
|
|
|
|
|
|
A minimal iOS app that generates images using Stable Diffusion v2.
|
|
|
|
|
You can create images specifying any prompt (text) such as "a photo of an astronaut riding a horse on mars".
|
|
|
|
|
|
|
|
|
|
The app uses
|
|
|
|
|
- macOS 13.0 or newer, Xcode 14.1
|
|
|
|
|
- iPhone 12+ / iOS 16.2+, iPad Pro with M1/M2 / iPadOS 16.2+
|
|
|
|
|
|
|
|
|
|
- stabilityai/Stable Diffusion v2 model, which was converted CoreML models using Apple's tool
|
|
|
|
|
- Apple / ml-stable-diffusion Swift Package (https://github.com/apple/ml-stable-diffusion#swift-requirements)
|
|
|
|
|
You can run the app on above mobile devices.
|
|
|
|
|
And you can run the app on Mac, building as a Designed for iPad app.
|
|
|
|
|
|
|
|
|
|
With the app, you can
|
|
|
|
|
The Xcode project does not contain the CoreML models of Stable Diffusion v2 (SD2).
|
|
|
|
|
So you need to make them converting the PyTorch SD2 models using Apple converter tools.
|
|
|
|
|
|
|
|
|
|
- try the image generation with Stable Diffusion v2 and Apple's Swift Package
|
|
|
|
|
- see how the Apple / ml-stable-diffusion Library works
|
|
|
|
|
The project uses the Apple/ml-stable-diffusion Swift Package.
|
|
|
|
|
You can see how it works through the simple sample code.
|
|
|
|
|
|
|
|
|
|
The project requires
|
|
|
|
|
- Apple/ml-stable-diffusion repo: https://github.com/apple/ml-stable-diffusion
|
|
|
|
|
|
|
|
|
|
- Xcode 14.1, macOS 13+
|
|
|
|
|
- iPhone 12+ iOS 16.2+ or iPad Pro/M1/M2 iPadOS 16.2+
|
|
|
|
|
![Image](images/ss1_240.png)
|
|
|
|
|
![Image](images/ss2_240.png)
|
|
|
|
|
|
|
|
|
|
Preparation
|
|
|
|
|
## Convert CoreML models
|
|
|
|
|
|
|
|
|
|
The coreml model files are too big to store in the GitHub repository. Git's file limitation is 100MB but the model files are total 2.5GB.
|
|
|
|
|
So the files were removed from the project.
|
|
|
|
|
You need to add the converted coreml model files yourself.
|
|
|
|
|
Convert the PyTorch SD2 model to CoreML models, following Apple's instructions.
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# create a Python environment and install dependencies
|
|
|
|
|
% conda create -n coremlsd2_38 python=3.8 -y
|
|
|
|
|
% conda activate coremlsd2_38
|
|
|
|
|
% cd SD2ModelConvChunked
|
|
|
|
|
% git clone https://github.com/apple/ml-stable-diffusion
|
|
|
|
|
% cd ml-stable-diffusion
|
|
|
|
|
pip install -e .
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Visit the Hugging Face Hub - stabilityai/stable-diffusion-2 model's page.
|
|
|
|
|
(https://huggingface.co/stabilityai/stable-diffusion-2)
|
|
|
|
|
Check the Terms and Use and accept it. Then you can use the model.
|
|
|
|
|
|
|
|
|
|
And you need a Hugging Face's `User Access Token`, to download huggingface/models.
|
|
|
|
|
Please visit Hugging Face's site and make an access token at Account Settings.
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# cli login
|
|
|
|
|
% % huggingface-cli login
|
|
|
|
|
Token: # input your Access Token
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Download and convert the SD2 PyTorch model to CoreML models.
|
|
|
|
|
If you do this on a Mac/8GB memory, please close all running apps except Terminal,
|
|
|
|
|
otherwise the converter will be killed due to memory issues.
|
|
|
|
|
|
|
|
|
|
Use these options:
|
|
|
|
|
- `--model-version stabilityai/stable-diffusion-2-base` ... model version
|
|
|
|
|
- `--bundle-resources-for-swift-cli` ... compile and output `mlmodelc` files into `<output-dir>/Resources` folder. The Swift Package uses them.
|
|
|
|
|
- `chunk-unet` ... split the Unet model into two chunks for iOS/iPadOS execution.
|
|
|
|
|
- `--attention-implementation SPLIT_EINSUM` ... use SPLIT_EINSUM for Apple Neural Engine(ANE).
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-text-encoder --convert-vae-decoder --convert-safety-checker -o sd2CoremlChunked --model-version stabilityai/stable-diffusion-2-base --bundle-resources-for-swift-cli --chunk-unet --attention-implementation SPLIT_EINSUM --compute-unit CPU_AND_NE
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Drag and drop the CoreML model files into `CoreMLModels` folder in the project.
|
|
|
|
|
|
|
|
|
|
- `merges.txt, vacab.json, UnetChunk2.mlmodelc, UnetChunk1.mlmodelc, VAEDecoder.mlmodelc, TextEncoder.mlmodelc`
|
|
|
|
|
|
|
|
|
|
![Image](images/ss3_240.png)
|
|
|
|
|
|
|
|
|
|
1. convert stabilityai/Stable-Diffusion-2-base PyTorch model to coreml models using Apple's tool.
|
|
|
|
|
2. add the files to the models2/Resources folder in the Xcode project.
|
|
|
|
|
- merges, TextEndoder, Unet, VAEDecoder, vocab
|
|
|
|
|
|
|
|
|
|
![Image](images/ss1_240.png)
|
|
|
|
|
![Image](images/ss2_240.png)
|
|
|
|
|
|
|
|
|
|
## Considerations
|
|
|
|
|
## Consideration
|
|
|
|
|
|
|
|
|
|
1. Chunked models: Chunked version, `UnetChunk1.mlmodelc` and `UnetChunk2.mlmodelc`, is better for iOS and iPadOS.
|
|
|
|
|
Follow the Apple's instructions. (https://github.com/apple/ml-stable-diffusion)
|
|
|
|
|
1. Large binary file: Since the model files are very large (about 2.5GB), it causes a large binary of the app.
|
|
|
|
|
- Large binary file: Since the model files are very large (about 2.5GB), it causes a large binary of the app.
|
|
|
|
|
The FAQ of Apple documentation says "The recommended option is to prompt the user to download
|
|
|
|
|
these assets upon first launch of the app. This keeps the app binary size independent of the
|
|
|
|
|
Core ML models being deployed. Disclosing the size of the download to the user is extremely
|
|
|
|
@ -45,5 +84,6 @@ important as there could be data charges or storage impact that the user might n
|
|
|
|
|
## References
|
|
|
|
|
|
|
|
|
|
- Apple Swift Package / ml-stable-diffusion: https://github.com/apple/ml-stable-diffusion
|
|
|
|
|
- Hugging Face Hub - stabilityai/stable-diffusion-2:(https://huggingface.co/stabilityai/stable-diffusion-2)
|
|
|
|
|
|
|
|
|
|
![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=flat)
|
|
|
|
|