Module 20
Local LLM Deployment
Run open-weights models on hardware you control, so private data never leaves the building. You start with Ollama on a laptop, pack it into Docker with a GPU on a server, give non-technical staff a private chat page with Open WebUI, and finish with a FastAPI service that other systems can call safely.
By the end of this module you can
- pick a local model that fits your hardware by working out its memory needs from parameters and quantization.
- run, customise and call models with Ollama through its native API, Python library and OpenAI-compatible endpoint.
- run Ollama in Docker with a persistent volume, GPU passthrough and Docker Compose, without exposing it to the network.
- set up Open WebUI for a team with accounts, knowledge collections and model presets, and know what you now have to secure and back up.
- build an async FastAPI service on Ollama with structured outputs, streaming, timeouts and a concurrency limit for one GPU.
Lessons
4 in this module- 01Ollama as a local runtimeYou can install Ollama, choose a model that fits your memory, call it through its native and OpenAI-compatible APIs, read its timing numbers, and bake settings into a Modelfile.
- 02Dockerised LLM environmentsYou can run Ollama in Docker with a volume that keeps models across rebuilds, pass an NVIDIA or AMD GPU through, and describe Ollama plus your app in one Docker Compose file.
- 03OpenWebUIYou can run Open WebUI against Ollama for a team, set up accounts, knowledge collections and model presets, and keep it private, backed up and honest about its answers.
- 04FastAPI services on top of OllamaYou can build an async FastAPI service on Ollama with schema-constrained JSON, a streamed endpoint, timeouts, a health check and a concurrency limit that keeps one GPU predictable.