Home Assistant next to Loxone with a local language model: what we built, what we did not and why

We are building Home Assistant next to a Loxone Miniserver in a new house. Loxone stays the controller, Home Assistant supplies what Loxone cannot, and a local language model is meant to serve the living room later without a word leaving the house. The state is a test: Home Assistant has run in a container since 14 October 2026, and the connection to Loxone is not set up yet. This post describes the rules that emerged while planning and measuring, and no finished house.

Why Home Assistant is needed at all

There is one reason, and it counts: Loxone has no radar sensor. The Presence Sensor Tree works with infrared and acoustics, and the older Motion Sensor Tree was pure infrared. A full-text search of the Loxone knowledge base for radar, mmWave and 24 GHz gave us no hit. Loxone's own documentation names the weakness. The property "presence duration without movement" has the default value 1800 seconds, and presence ends when no movement has been detected for that duration. That prevents ambient noise from keeping presence active unintentionally. It also means that anyone sitting still drops out, and anyone who has left with the radio still on stays in.

For this Home Assistant reads mmWave sensors that also detect people without movement. If the reason disappears because Loxone brings a radar sensor itself, the whole setup disappears. We do not want to keep building it out of habit.

The rule: sensor values go in, logic stays in Loxone

The sensor value goes to Loxone through a virtual input. The logic does not live in Home Assistant. (Quelle: Our own plan, as of 29 Aug 2026)

The direction is not arbitrary. PyLoxone issue 511 shows that a write from Home Assistant to the Lighting Controller V2 fails silently, and the switch falls back after two seconds. And a restart of Home Assistant must not switch off the light in the guest toilet. For the route into the virtual input a rest_command from Home Assistant is therefore enough, and PyLoxone is not needed for it at all.

We read the PV system and the heat pump through Loxone and not through a second Modbus client in Home Assistant. The PV system's dongle tolerates only two simultaneous connections.

Two buying traps with the sensors

The 15-euro Zigbee sensor is a purchase lottery. The name "ZY-M100" stands for a family: Zigbee2MQTT lists seven model entries which differ only in the Tuya manufacturerName, invisible before purchase. A new batch with an unknown ID ends as "Unsupported device". We therefore rely on ESPHome with the LD2450 sensor, which can do multi-target and zone detection. The component is part of ESPHome itself, with no extra package. mmWave needs constant power anyway, and a Zigbee mesh would only add a second network.

The repositories of the popular Everything Presence devices carry no LICENSE file, with several hundred stars. Anyone running a device privately is unaffected. Anyone who takes over, changes or commercially uses the code has no right to use it.

The local language model

The goal is a voice assistant in the living room like Google Assistant, completely local. The speech chain in the test consists of Home Assistant, Piper (voice de_DE-thorsten-medium) and Speech-to-Phrase for German. As the model machine we planned the AMD R9700 with 32 GB, which we removed from another server. It is a small machine of its own in the house, with no Proxmox, because one workload needs no hypervisor.

We measured when choosing the engine. The measurement ran on the same card with the same model, Qwen3.8-27B, 256 tokens per request, on 27 August 2026.

Our own measurement of 27 Aug 2026, Qwen3.8-27B, 256 tokens per request. The lead at one request comes from the MTP draft head, which vLLM cannot use on this architecture. Without it llama.cpp makes 25.3 tokens per second. (Quelle: Our own measurement, 27 Aug 2026)

A home assistant is one stream, so the first column counts: 36.9 against 26.1 tokens per second. We take llama.cpp. ROCm is also the only usable route, because Vulkan throws the card into a page fault on RDNA4.

Two points cost time or would have:

A 27B model that thinks is the wrong tool for "light on". With speech recognition, prefill and speech output we expect three to five seconds, against about one second at Google. That is an estimate and no measured value. That is why Home Assistant is set to "handle commands locally first": device commands hit an intent and run in under a second, and the model only gets real questions.
Without enable_thinking: false the model returns message.content empty, with everything in reasoning_content, with HTTP 200 and no error. That has already happened to us twice in the homelab.

Traps in the Home Assistant test

Piper reports the language de_DE. Anyone who writes de or de-DE into the pipeline gets "Language not supported".
On first setup Home Assistant creates a Google Translate TTS service. It sends the text to Google. We deleted it and check after every new setup whether it is back.
Home Assistant's MCP server only limits what is visible through MCP. The long-lived token is an ordinary user token and opens the whole interface of that user. Access therefore belongs only with a user of its own with as few rights as possible, after emptying the exposed entities, and Home Assistant does not belong on the internet. The integration also offers an option to restrict the endpoint to administrator accounts, and according to the documentation clients only see entities exposed to them.
After the house network moved, Loxone devices entered by hostname stopped working. In Loxone Config IP addresses belong in, and DHCP reservations, because names are not resolved in the new network.

What is open

PyLoxone is installed but not set up yet, and the virtual input in Loxone is missing. The test runs for now with a test access. Before real operation Home Assistant needs a user of its own in Loxone with minimal rights. The microphone in the room is not bought yet, and the plan is the Home Assistant Voice Preview Edition, about 60 euros per room. PyLoxone essentially has one developer (559 commits against 35 for the second), but is licensed under Apache 2.0 and has no alternative, since the next candidate has six stars against 304.

How we worked

This post rests on our own planning and measurement notes from 29 August to 14 October 2026, on the Home Assistant documentation (MCP server) and on the PyLoxone and ESPHome repositories. The llama.cpp and vLLM measurement is ours, and its raw data is with us. The "three to five seconds" is an estimate. The setup in the living room does not exist yet.