Andrew Ng’s OpenWorker: Local AI Agent That Returns Finished Work

Andrew Ng has released OpenWorker, an MIT-licensed open-source desktop AI agent designed to return completed work rather than conversational replies. The app runs its full four-layer stack locally on the user's machine, supports 30 curated models via bring-your-own-key, and gates every action through a typed risk engine. It spans 119 Python files and 149 TypeScript/TSX files, and it is available now on GitHub alongside a project site.

What happened

Detail	Fact
License	MIT
Python codebase	119 files, ~32,400 lines under coworker/
Frontend files	149 TypeScript/TSX files under surfaces/gui/
Backend test modules	78
Local agent port	127.0.0.1:8765
Turn cap	12 model-tool iterations (example config)
Supported models	30 curated entries

OpenWorker asks the user to define an outcome, not write a prompt. You tell it you want a polished document, a triaged inbox, or a calendar update. It breaks that goal into steps, works across local files and connected apps, and pauses for your approval before anything consequential happens.

How the architecture works

The whole stack sits on four layers, all running on the user’s device:

1. Desktop shell: A Tauri 2 native window wrapping a React 18 UI (bundle ID: com.openworker.desktop). The shell also supervises the Python server process.
2. Local agent server: Python 3.10+ on FastAPI and uvicorn, binding to 127.0.0.1:8765. The default turn cap is 12 model-tool iterations.
3. Capability and connector layer: Local tools covering files, git, ripgrep-backed search, shell, and to-do, plus hosted integrations and MCP (Model Context Protocol, a standard for giving AI agents access to external data sources).
4. Model router: A single interface over native, OpenAI-compatible, reseller, and local providers. The engine is built on aisuite, Ng’s own provider-agnostic LLM library.

Which models can you use?

There is no OpenWorker inference service. Users paste an API key or point the app at a local runtime. The 30-model matrix covers:

• Native providers: OpenAI (GPT-5.6 Sol, Terra, Luna and GPT-5.5), Anthropic (Claude Fable 5, Opus 4.8, Sonnet 4.6, Haiku 4.5), Google (Gemini 3.1 Pro, 3.6 Flash, 2.5 Pro, 2.5 Flash).
• OpenAI-compatible vendors: GLM-5.2, DeepSeek V4, Kimi K2.6, MiniMax M2.5, Qwen3 Max, Grok 4.3, Mistral Large.
• Hosted open-weight models: Via Together AI and Fireworks.
• Fully local models: Via Ollama, which requires no API key at all.

Why the permission engine matters

Most desktop agent projects bolt on an approval dialog as an afterthought. OpenWorker treats permissions as a typed layer at the core of the system. Every tool call falls into one of four risk classes:

• read: No side effects.
• write_local: Mutates the workspace, path-scoped.
• exec: Runs shell commands.
• external: Has side effects off the machine.

Five permission modes control what happens: discuss and plan are read-only; interactive is the default and asks before writes, commands, and external actions; auto allows everything while remaining path-scoped; custom auto-approves a user-specified set of tools.

Two design decisions stand out. First, unattended mode does not raise the autonomy ceiling. It only changes where the human gets reached: approvals route to an Inbox and the session suspends until answered. Second, shell commands always ask, by design. Task-scoped standing rules are restricted to the external risk tier only.

The built-in ops persona also instructs the model to treat content from tools, logs, files, and incoming messages as untrusted data rather than instructions. That is an explicit defense against prompt injection (attacks where malicious content in an external file tricks the agent into taking unintended actions), written directly into the shipped persona.

What about privacy?

Model calls go directly from your machine to the provider you configured. Conversations, connector tokens, and model keys stay local. The secret store is designed so that secrets never enter model context, prompts, or traces. The only optional cloud component is a broker that handles OAuth handshakes for one-click connectors, using Auth0 Authorization Code with PKCE. Connector tokens are handed straight to the machine and never stored in the cloud. The app works fully offline when you paste credentials manually.

Why it matters

The gap between “AI that chats” and “AI that ships work” is where most current tools fall short. OpenWorker is a serious attempt at closing it, built by someone with the credibility and resources to attract contributors fast. The local-first architecture matters for businesses that cannot send internal files or credentials to a cloud agent. The growing range of capable open-weight models also means the Ollama path is increasingly practical, not just a fallback.

For teams already experimenting with AI integration inside their workflows, OpenWorker is worth watching closely. A typed permission model and an explicit prompt-injection posture are two features that most commercial agent products have not shipped yet.

Our take

The permission architecture is the real story here, not the model list. Typed risk classes with a built-in unattended inbox and a hard rule that shell commands always ask is the kind of design discipline that separates a prototype from something you can actually trust with your calendar or codebase. The prompt-injection defense baked into the system persona is a detail that most teams building agents skip entirely, because it is unglamorous and hard to test. Ng shipping it as a default is a good signal.

The caveat: 30 models, some with version numbers that do not exist in public documentation yet (GPT-5.6, Claude Fable 5), suggests the model list reflects an internal roadmap rather than what is available today. Treat it as a target architecture and test with whatever models are actually accessible to you right now. If you are considering building something similar for your own operations, see what workflow automation already handles before adding an agent layer on top.

What to do about it

1. Check the GitHub repo and confirm which models in the 30-entry list are currently available via your existing API keys.
2. Run it against a low-stakes task (inbox triage, a draft document) in interactive mode before touching anything write_local or exec.
3. Read the built-in ops persona source to understand exactly what prompt-injection protections are in place before connecting sensitive apps.
4. If you want fully private operation, test the Ollama path with a capable open-weight model before committing to a hosted provider.

Start in interactive permission mode, verify the model list against live API availability, and treat the permission engine as the feature worth evaluating most carefully.

Source: Marktechpost