Voice and AI

AI in Home Assistant: Gemini, OpenClaw and the Best Local AI Models

Connect Gemini or a local AI model to Home Assistant, pick one that fits your hardware, and understand OpenClaw and its risks before you install it.

On this page
  1. How AI fits into Home Assistant’s voice setup
  2. Two things called Gemini
  3. How to get Gemini on Home Assistant
  4. Google Assistant and Gemini: what’s changing
  5. Running a local model with Ollama
  6. Choosing hardware and model size
  7. Small fine-tuned models: Home-LLM
  8. OpenClaw: what it is and how to approach it
  9. Comparing the options
  10. Common misconceptions
  11. Safety

How AI fits into Home Assistant’s voice setup

Home Assistant’s built-in conversation agent handles simple commands, such as “turn off the hall light,” without any AI. An AI agent takes over for requests the built-in agent does not understand, such as questions about your home or more open-ended instructions. If a phrase you use often is simple, a custom voice command can handle it without any AI at all.

Official LLM conversation integrations include OpenAI, Google Generative AI (Gemini), Anthropic, Ollama and OpenRouter. The list grows with each release. Two settings matter most:

  • Control Home Assistant: set this to Assist so the model can control the entities you have exposed. Without it, the model can chat but cannot switch anything.
  • Prefer handling commands locally: built-in intents handle simple commands first, and only other requests go to the model. This is faster and cheaper, because simple commands skip the API.

Two things called Gemini

Searching for “Home Assistant Gemini” turns up two different products:

  • Gemini for Home is Google’s AI assistant. It is replacing Google Assistant on Nest speakers and phones. You control it through Google’s own apps, not through Home Assistant.
  • Gemini in Home Assistant is the official Google Generative AI integration. It connects a Gemini model to Home Assistant as a conversation agent, and you set it up with a Google API key.

This article is about the second one.

How to get Gemini on Home Assistant

  1. Get an API key. Create a free API key in Google AI Studio. Flash models are the usual pick for voice. Model names change often, so check the list in the integration’s options.
  2. Add the integration. Go to Settings > Devices & services > Add integration, choose Google Generative AI, and paste your API key.
  3. Choose it as the conversation agent. Go to Settings > Voice assistants, open or create an assistant, and set its conversation agent to the Gemini integration.
  4. Set control and exposure. Set Control Home Assistant to Assist, and expose the entities you want the model to see under Settings > Voice assistants > Expose.

Free-tier Gemini has rate limits, and Google may use free-tier data to improve its products. If that matters to you, read Google’s current terms for the tier you use before you set it up.

Google Assistant and Gemini: what’s changing

Google is replacing Google Assistant with Gemini on Nest speakers and phones. The Home Assistant Google Assistant integration still works, but the steps in the Google Home app may look different from older guides. For the Google Assistant connection itself, see how to add Google Assistant to Home Assistant.

Running a local model with Ollama

Ollama is the usual way to run a local model for Home Assistant. Connect Home Assistant to your Ollama server through the Ollama integration, then pick a model. The model must support tool calling, or it cannot control devices. Not every Ollama model does, so check before you download one.

Two settings matter most for a local model:

  • Context window: raise Ollama’s num_ctx to 8k or more. The default is too small for many exposed entities, so the model may not see all of them.
  • Exposure: expose only the entities the model needs. Each request carries the entity list, and a long list uses more of the context window.

Speech recognition and satellites are covered in Home Assistant voice control.

Choosing hardware and model size

Hardware is the deciding factor. Rough guidance for quantized (4-bit) models:

  • 7–8B models need about 6–8 GB of VRAM.
  • 14B models need about 12–16 GB.
  • CPU-only setups are too slow for voice, even with a small model.

Good starting picks for 2025–26 include Qwen 2.5 or Qwen3 at 7–8B, Llama 3.1 8B and Ministral 8B. This list changes fast, so treat it as a starting point rather than a ranking.

Latency matters more than size for voice. A bigger model that takes several seconds to answer can feel worse than a smaller one that answers quickly. Test with your real devices before you settle on one.

Small fine-tuned models: Home-LLM

If your hardware is limited, a small model fine-tuned for home control may be enough. Home-LLM, a HACS integration by acon96, offers small models such as Home-3B for low-power hardware. These models are built for controlling a home rather than for general chat.

OpenClaw: what it is and how to approach it

OpenClaw, formerly known as Clawdbot and Moltbot, is an open-source AI agent that you run yourself. You use it through chat apps, and it can call the Home Assistant API. It is not an official Home Assistant integration.

The details of how OpenClaw connects to Home Assistant change quickly, and community projects differ from one another. Check the current project documentation before you follow any setup guide.

If you do try it:

  • Use a dedicated Home Assistant user, not your admin account, and create a long-lived access token for that user.
  • Review community skills before you install them. Malicious OpenClaw skills have been reported.
  • Think about what it can reach. Community members note that add-ons and integrations can read files in /config, such as configuration.yaml, and may have system-level access if you grant it.
  • Keep locks, alarms and garage doors out of its reach.

Comparing the options

Option Where it runs Good for Watch out for
Gemini integration Google’s cloud Quick setup, no local hardware Free-tier rate limits and data use
OpenAI integration OpenAI’s cloud Small, quick, low-cost models API billing, separate from ChatGPT Plus
Ollama with a 7–8B model Your hardware Privacy and no per-request fee 6–8 GB of VRAM, and tool calling is required
Home-LLM small models Your hardware Low-power hardware Built for home control, not general chat
OpenClaw Your own hardware or server Chat-based control and automation Not official, and it needs careful access control

If you prefer OpenAI, see how to connect ChatGPT to Home Assistant.

Common misconceptions

  • An LLM is not needed for basic voice control. Built-in intents handle common commands.
  • Bigger is not always better. For voice, latency matters as much as quality.
  • Not every Ollama model can control devices. The model needs tool calling.
  • Consumer subscriptions do not include API access. ChatGPT Plus, for example, is separate from the API.

Safety

  • Do not expose locks, alarms or garage doors to an LLM. A model can misread a request.
  • Review community OpenClaw skills before you install them, and treat each new skill as code that runs in your home.

The complete guide to voice and ai

Home Assistant Voice Control: Wake Words and Local Voice Assistants
Read the guide

Keep exploring

Guides from other topics

All topics →