Generative Artificial Intelligence in Spring Boot 3.2 using SD4J

Disclaimer : This is a combination of my Spring Boot knowledge/expertise and original post by Johan Janssen. SD4J was written by Oracle and I have made use of that in my Spring Boot application. Original post and link to SD4J source code are available in the references section. All rights and credits are given to respective first authors.

Generative Artificial Intelligence is taking over the world by storm. From creative Poem writing to Advanced computer code generation and everything in between can be done using GenAI nowadays. Another one interesting usage of this technology is Text-To-Image feature where a sentence can be used to generate an image. In the following example you can see an image generated from the phrase “Rose”.

Oracle Open Source recently introduced their open source project Stable Diffusion in Java (SD4J), which is a modified port of the Stable Diffusion C# implementation with support for negative text inputs. Stable diffusion is a deep learning text-to-image model based on diffusion. ONNX Runtime, a cross platform inference and training machine learning accelerator is required to run SD4J.

Setting Up (MacBook Pro M1)

You would need Git Large File Storage, a Git extension for versioning large files, should be installed first, for example with the following command on Mac M1:

$ sudo brew install git-lfs

Afterwards, the SD4J project can be cloned locally with the following command:

$ git clone https://github.com/oracle-samples/sd4j

SD4J uses models, the compatible pre-built ONNX models from Hugging Face, that will be used for the examples in this news story:

$ git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 -b onnx

ONNXRuntime-Extensions is a library which extends the capabilities of the ONNX models and the interference with the ONNX Runtime:

$ git clone https://github.com/microsoft/onnxruntime-extensions

After cloning the project, the following command can be executed inside the onnxruntime-extensions directory to compile the ONNXRuntime-Extensions for your platform:

$ env /usr/bin/arch -x86_64 /bin/zsh --login

$ ./build_lib.sh --config Release --update --build --parallel

The following error might be displayed if CMake isn’t installed:

[ERROR] - Unable to find CMake executable. Please specify its path with --cmake_path.

Install at least version 3.25 of CMake to resolve the error, for example with the following command on Mac M1:

$ sudo brew install cmake

When the build is successful, the resulting library (libortextensions.dylib) can be found inside the following directory:

build/<OS-name>/Release/lib/

Running

Clone the https://github.com/shazin/genai-demo project. Afterwards copy the libortextensions.dylib file into the root of cloned project.

You can then run the GenaiDemoApplication main class with following application argument

--model-path <Absolute Path to>/stable-diffusion-v1-5 --execution-provider CPU

and following VM argument if you are using Java 21 and want to enable Spring Boot 3.2 to use Virtual Threads.

-Dspring.threads.virtual.enabled=true

When the application is running you can send a request via a browser like following http://localhost:8080/stablediffusion?q=Wildlife%20photograph%20of%20an%20astronaut%20riding%20a%20horse%20in%20the%20desert

The search phrase is

“Wildlife photograph of an astronaut riding a horse in the desert”

Which will show activities in the terminal of the running application.

Eventually it will show the generated image in the browser.

This demo shows the power of open source implementation of Stable Diffusion and how we can make use of it from within a Spring Boot application which is my expertise.

References

Stable Diffusion in Java (SD4J) Enables Generating Images with Deep Learning

M1 Mac — Switching Terminal between x86_64 and arm64

https://github.com/oracle-samples/sd4j