updated README.

main
Yasuhito Nagatomo 2 years ago
parent bbd4e12337
commit d584ba875d

@ -2,41 +2,80 @@
![AppIcon](images/appicon180.png) ![AppIcon](images/appicon180.png)
![Image](images/ss0_1280.png)
A minimal iOS app that generates images using Stable Diffusion v2. A minimal iOS app that generates images using Stable Diffusion v2.
You can create images specifying any prompt (text) such as "a photo of an astronaut riding a horse on mars".
The app uses - macOS 13.0 or newer, Xcode 14.1
- iPhone 12+ / iOS 16.2+, iPad Pro with M1/M2 / iPadOS 16.2+
- stabilityai/Stable Diffusion v2 model, which was converted CoreML models using Apple's tool You can run the app on above mobile devices.
- Apple / ml-stable-diffusion Swift Package (https://github.com/apple/ml-stable-diffusion#swift-requirements) And you can run the app on Mac, building as a Designed for iPad app.
With the app, you can The Xcode project does not contain the CoreML models of Stable Diffusion v2 (SD2).
So you need to make them converting the PyTorch SD2 models using Apple converter tools.
- try the image generation with Stable Diffusion v2 and Apple's Swift Package The project uses the Apple/ml-stable-diffusion Swift Package.
- see how the Apple / ml-stable-diffusion Library works You can see how it works through the simple sample code.
The project requires - Apple/ml-stable-diffusion repo: https://github.com/apple/ml-stable-diffusion
- Xcode 14.1, macOS 13+ ![Image](images/ss1_240.png)
- iPhone 12+ iOS 16.2+ or iPad Pro/M1/M2 iPadOS 16.2+ ![Image](images/ss2_240.png)
Preparation ## Convert CoreML models
The coreml model files are too big to store in the GitHub repository. Git's file limitation is 100MB but the model files are total 2.5GB. Convert the PyTorch SD2 model to CoreML models, following Apple's instructions.
So the files were removed from the project.
You need to add the converted coreml model files yourself. ```bash
# create a Python environment and install dependencies
% conda create -n coremlsd2_38 python=3.8 -y
% conda activate coremlsd2_38
% cd SD2ModelConvChunked
% git clone https://github.com/apple/ml-stable-diffusion
% cd ml-stable-diffusion
pip install -e .
```
Visit the Hugging Face Hub - stabilityai/stable-diffusion-2 model's page.
https://huggingface.co/stabilityai/stable-diffusion-2
Check the Terms and Use and accept it. Then you can use the model.
And you need a Hugging Face's `User Access Token`, to download huggingface/models.
Please visit Hugging Face's site and make an access token at Account Settings.
```bash
# cli login
% % huggingface-cli login
Token: # input your Access Token
```
Download and convert the SD2 PyTorch model to CoreML models.
If you do this on a Mac/8GB memory, please close all running apps except Terminal,
otherwise the converter will be killed due to memory issues.
Use these options:
- `--model-version stabilityai/stable-diffusion-2-base` ... model version
- `--bundle-resources-for-swift-cli` ... compile and output `mlmodelc` files into `<output-dir>/Resources` folder. The Swift Package uses them.
- `chunk-unet` ... split the Unet model into two chunks for iOS/iPadOS execution.
- `--attention-implementation SPLIT_EINSUM` ... use SPLIT_EINSUM for Apple Neural Engine(ANE).
```bash
python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-text-encoder --convert-vae-decoder --convert-safety-checker -o sd2CoremlChunked --model-version stabilityai/stable-diffusion-2-base --bundle-resources-for-swift-cli --chunk-unet --attention-implementation SPLIT_EINSUM --compute-unit CPU_AND_NE
```
Drag and drop the CoreML model files into `CoreMLModels` folder in the project.
- `merges.txt, vacab.json, UnetChunk2.mlmodelc, UnetChunk1.mlmodelc, VAEDecoder.mlmodelc, TextEncoder.mlmodelc`
![Image](images/ss3_240.png)
1. convert stabilityai/Stable-Diffusion-2-base PyTorch model to coreml models using Apple's tool.
2. add the files to the models2/Resources folder in the Xcode project.
- merges, TextEndoder, Unet, VAEDecoder, vocab
![Image](images/ss1_240.png)
![Image](images/ss2_240.png)
## Considerations ## Consideration
1. Chunked models: Chunked version, `UnetChunk1.mlmodelc` and `UnetChunk2.mlmodelc`, is better for iOS and iPadOS. - Large binary file: Since the model files are very large (about 2.5GB), it causes a large binary of the app.
Follow the Apple's instructions. (https://github.com/apple/ml-stable-diffusion)
1. Large binary file: Since the model files are very large (about 2.5GB), it causes a large binary of the app.
The FAQ of Apple documentation says "The recommended option is to prompt the user to download The FAQ of Apple documentation says "The recommended option is to prompt the user to download
these assets upon first launch of the app. This keeps the app binary size independent of the these assets upon first launch of the app. This keeps the app binary size independent of the
Core ML models being deployed. Disclosing the size of the download to the user is extremely Core ML models being deployed. Disclosing the size of the download to the user is extremely
@ -45,5 +84,6 @@ important as there could be data charges or storage impact that the user might n
## References ## References
- Apple Swift Package / ml-stable-diffusion: https://github.com/apple/ml-stable-diffusion - Apple Swift Package / ml-stable-diffusion: https://github.com/apple/ml-stable-diffusion
- Hugging Face Hub - stabilityai/stable-diffusion-2:https://huggingface.co/stabilityai/stable-diffusion-2
![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=flat) ![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=flat)

Binary file not shown.

After

Width:  |  Height:  |  Size: 960 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

Loading…
Cancel
Save