postgrok

local-ai-threat-layers

Properties3
tagssecurity, local-llms, supply-chain
typeconcept
statusseedling

4 min read

Local AI threat layers

Local AI security is not a binary. The Kai video gives the working decomposition, five layers, each with its own threat and its own mitigation, and the argument rests on one sentence about where risk lives: “The risk isn’t about where the model runs, it is about what the model can reach” (22:22).

  1. The file. Weights in a GGUF container. Not executable. A JPEG cannot open a terminal, and the sha-256 you check only proves you received what the uploader meant to send, silence on what they meant. Scanners (picklescan, modelscan, the hub’s built-in checks) answer one question: does this file run code on load, the 2024 pickle attack, the thing SafeTensors was invented against.
  2. The chat template. Jinja embedded in every GGUF, rendered on every prompt, code by any definition, unseen by almost everyone. It routes both ways: unsandboxed rendering is remote code execution (the 2024 llama-cpp-python finding, Checkmarx; CVE-2026-5760 in SGLang, CVSS 9.8, April 2026), and even fully sandboxed it is an instruction channel, since trigger-phrase backdoors planted at this layer cut accuracy from 90 to 15 percent while the hub’s automated scans passed the files (pillar-security across 18 models and four engines). The template can differ between the model card display and the downloaded file.
  3. The runner. Ollama, LM Studio, llama.cpp server, vLLM, SGLang. Programs that parse files written by strangers, the historical home of memory-safety bugs: heap reads via forged tensor sizes (Bleeding Llama, below Ollama 0.17.1), an auto-updater whose signature check unconditionally returned success, a 2024 path traversal from download to RCE (Wiz). Ollama gets the headlines because it is the most popular runner; the category is the problem.
  4. The port. The default is loopback, which is why the exposure studies matter: SentinelOne and Censys counted 175,000 unauthenticated Ollama hosts answering the public internet over 293 days, more than 48 percent advertising tool calling; a Nankai University longitudinal scan (2609.07115) found 152,137 cumulative exposed IPs Feb 2025 to Feb 2026. OLLAMA_HOST=0.0.0.0 is one environment variable from “personal” to “world”, and Docker’s published ports bypass the firewall you configured on purpose. DNS rebinding walks the browser inside your network, and the host-header defense that would stop it is skipped exactly when bound to 0.0.0.0. A wrapper that binds wide while printing “localhost” (Nvidia’s NemoClaw, August 2026) closes the loop: config lies, scanners miss.
  5. Weights plus tools. The layer where the model stops being a file and becomes an actor. projectdiscovery poisoned 125 of 625 rows of Qwen2.5-7B tool-calling data, roughly $8 to 2.5 hours on one rented GPU, and got a backdoor that sends .env to a fetched payload on the trigger “bonsoir, Elliot”: 50 fires in 50, normal calls at 99 to 100 percent, weights scan-clean because the weights are clean. The payload lives at a URL the attacker controls afterward. Defenses fail in a pattern: allow-listing does not stop it (the domain is trusted), benchmarks cannot test an unknown trigger, and safety fine-tuning does not reliably scrub these, which is the finding Kai reports from a 2024 paper he attributes to Anthropic; the closest verified study this session is anthropic’s 2025-10 work with the UK AI Security Institute (about 250 poison documents across 600M to 13B parameters, success tracking the absolute count rather than the ratio), and the 2024 attribution stays open.

What follows

A setup is safe relative to its layers. Chat-only, official weights, current runner, loopback: local remains the private option, and his answer for that regime is to go enjoy it. The moment you add tools, the moment you pull a stranger’s template because a community maintains it and it is on version 22, the moment you bind to 0.0.0.0 so a phone can query the desktop, the threat model changes and nothing in the file told you. His five fixes are the cheapest ordering of the layers: loopback plus a tunnel (Tailscale), updated runner, official weights for anything with tools (“an obliterated merge from a random account with 2 million downloads is a pull request from a stranger”, 23:25), secrets outside the working directory, and privilege separation between the process that executes and the process that has network.

The vault’s angle: this is environment-and-constraints with the sign flipped. The same claim that makes agents productive, that a machine-checked environment catches what judgment misses, makes a poisoned environment decisive, because the agent will trust exactly the channels you told it to. The supply-chain reading belongs beside phantom-citations and technological-lock-in. The model card showing a clean template while the download carries a different one is a provenance failure in the verifiability-and-one-way-doors sense: the artifact and its label are not one object.

Sources

Graph View

Backlinks