Photo by Marius Masalar on Unsplash

Building a Persistent Local AI Stack

Composed

Installing and running local AI models on Linux used to feel far more intimidating than it does now. With current tools, it is possible to build a local setup that is not only easier to deploy, but also easier to maintain over time. With the use of containers, you can keep your models, chats, and related data organized while avoiding the fragility of an ad hoc install.

In this article, I show you how to build a streamlined, fully local AI stack that works even on complex partition layouts and remains reliable through routine updates. With Ollama on the back end and Open WebUI on the front end, you will create a private, web-based assistant that you can adapt to your workflow. I will also show you how Docker Compose can manage installation and updates for both services.

My build and setup was tested on both Ubuntu 24.04 with no GPU and Ubuntu 25.10 with an NVIDIA GPU. Therefore, I have provided the instructions for both configurations. Along the way, I cover the Compose layout, persistent storage, restart behavior, safe update habits, backup and restore planning, and rollback basics and provide a few brief notes on model choice for less powerful systems.

What to Build

At the core of the build is a local AI stack of Ollama and Open WebUI. Ollama is a powerful runtime for local AI models, providing a standardized API for large language models (LLMs) and related workloads. However, Ollama only provides a command-line interface (CLI), which can be limiting for some users. Open WebUI adds a web-based interface that connects to the Ollama API and provides access to additional tools, user management, chat history, and other features.

To tie these two services together so they can be started, stopped, updated, and recreated as a single application stack, you can use Docker Compose, which allows you to define a complex stack with a single YAML file. This arrangement is useful for more