Running large language models (LLMs) and other generative AI models can be a complex, frustrating process of managing dependencies, drivers, and environments. At Docker, we believe this should be as simple as docker model run.
That’s why we built Docker Model Runner, and today, we’re thrilled to announce a new collaboration with Universal Blue. Thanks to the fantastic work of these contributors, Docker Model Runner is now included in OSes such as Aurora and Bluefin, giving developers a powerful, out-of-the-box AI development environment.
What is Docker Model Runner?
For those who haven’t tried it yet, Docker Model Runner is our new “it just works” experience for running generative AI models.
Our goal is to make running a model as simple as running a container.
Here’s what makes it great:
- Simple UX: We’ve streamlined the process down to a single, intuitive command: docker model run <model-name>.
- Broad GPU Support: While we started with NVIDIA, we’ve recently added Vulkan support. This is a big deal—it means Model Runner works on pretty much any modern GPU, including AMD and Intel, making AI accessible to more developers than ever.
- vLLM: Perform high-throughput inference with an NVIDIA GPU
The Perfect Home for Model Runner
If you’re new to it, Universal Blue is a family of next-generation, developer-focused Linux desktops. They provide modern, atomic, and reliable environments that are perfect for “cloud-native” workflows.
As Jorge Castro who leads developer relations at Cloud Native Computing Foundation explains, “Bluefin and Aurora are reference architectures for bootc, which is a CNCF Sandbox Project. They are just two examples showing how the same container pattern used by application containers can also apply to operating systems. Working with AI models is no different – one common set of tools, built around OCI standards.”
The team already ships Docker as a core part of its developer-ready experience. By adding Docker Model Runner to the default installation (specifically in the -dx mode for developers), they’ve created a complete, batteries-included AI development environment.
There’s no setup, no config. If you’re on Bluefin/Aurora, you just open a terminal and start running models.
Get Started Today
If you’re running the latest Bluefin LTS, you’re all set when you turn on developer mode. The Docker engine and Model Runner CLI are already installed and waiting for you. Aurora’s enablement instructions are documented here.
You can run your first model in seconds:
This command will download the model (if not already cached) and run it, ready for you to interact with.
If you’re on another Linux, you can get started just as easily. Just follow the instructions on our GitHub repository.
What’s Next?
This collaboration is a fantastic example of community-driven innovation. We want to give a huge shoutout to the greater bootc enthusiast community for their forward-thinking approach and for integrating Docker Model Runner so quickly.
This is just the beginning. We’re committed to making AI development accessible, powerful, and fun for all developers.
How You Can Get Involved
The strength of Docker Model Runner lies in its community, and there’s always room to grow. We need your help to make this project the best it can be. To get involved, you can:
- Star the repository: Show your support and help us gain visibility by starring the Docker Model Runner repo.
- Contribute your ideas: Have an idea for a new feature or a bug fix? Create an issue to discuss it. Or fork the repository, make your changes, and submit a pull request. We’re excited to see what ideas you have!
- Spread the word: Tell your friends, colleagues, and anyone else who might be interested in running AI models with Docker.
We’re incredibly excited about this new chapter for Docker Model Runner, and we can’t wait to see what we can build together. Let’s get to work!
