Building a Personal AI Agent Stack with Claude Code
Home Linux server + Proxmox virtualization → multi-VM isolated architecture → build personal tools for real-life problems one by one
- Who
- Individuals with Linux and self-hosting experience who want to use AI to solve real personal problems
- You get
- A multi-layer agent system with a coordinator, isolated coaches, quick assistants, and specific apps, core development completed in just over two weeks
- You need
- Claude Code · GitHub · Telegram
Constraints
Requires an always-on Linux host (recommended 10+ cores, 32 GB RAM), Proxmox virtualization, Claude Code subscription, and basic Linux networking and permissions management. Costs include one-time hardware and model API calls (author reports ~$20/mo Claude Pro plus minor OpenRouter fees). All code currently resides in private repositories and must be implemented independently.
What they did
Earlier this year the author assembled a Linux PC for photo storage and backups, then decided to also use it as an always-on personal AI assistant server. After sitting boxed for months, he went all-in over several weeks, using Claude Code (primarily Opus 5.5) to build a multi-layer personal agent stack: a Chief of Staff coordinates cross-machine work, isolated Build Sessions keep each component's context focused, Coaches handle long-running single-responsibility tasks, Quick Assistants handle short conversations, and Dashboards handle screen-based work with history. Specific apps included a coach for his son's exam prep, LifeVault family records, Recall recommendation aggregation, a Career job-search pipeline, Mail Watch email flagging, and a cost dashboard.
How it went
The author reports the core system came together in just over two weeks. Claude Code proved effective not only at writing code but also operating infrastructure: configuring services, creating VMs, and maintaining containers. The Chief of Staff completed a security audit and coordinated patching across machines in a few hours. Recall testing showed DeepSeek V4.1 Flash extracted 98 items from the author's data at about one-third the cost of Gemini, but DeepSeek was excluded from LifeVault because it occasionally missed statement dates. The author emphasizes low-risk reversible tasks need less supervision, while changes involving data, networking, or external systems require explicit review and approval gates.
Worth knowing
1. Security boundaries should come from architecture, not just prompts: the agent handling untrusted web/email content has no tools, memory, or personal context, returning structured data for code validation. 2. Evaluate models on your actual workload: the same model can perform very differently across tasks, and the cost of being wrong matters as much as generation cost. 3. Memory and context need active management: prune CLAUDE.md / AGENTS.md periodically, cap session size, use /wrap-up to preserve decisions and unfinished tasks rather than full history; shared memory should only hold one-sentence facts expected to remain true for months. 4. An agent should not be the only recovery path from its own changes: cross-machine changes require documented rollback steps, and critical changes need recovery mechanisms that work even if the agent loses access. 5. Code for rules, models for interpretation: hard constraints are filtered by code before models evaluate remaining candidates, reducing decisions left to models. 6. Voice input (the author uses Wispr Flow) can significantly speed interaction with coding agents, though tool stability varies. 7. All code currently lives in private repositories; personal data stays out of Git, and secret management is still evolving. 8. The author explicitly distinguishes personal experimentation from production software: personal tools can tolerate some mess provided blast radius is limited and basic hygiene is maintained.
Steps
Prepare hardware and virtualization environment
Assemble or prepare an always-on Linux PC: the author used an Intel i5-14400 (10 cores, 16 threads), 32 GB RAM, NVMe SSD for the system, and two 8 TB drives in ZFS mirror for photos and backups. Install Proxmox and create separate environments: development, apps, agents, Hermes, and local photo storage and backups.
Set network and security baseline
Do not expose home-lab hosts directly to the internet; let them initiate outbound connections instead. Connect Windows desktop directly to the Linux PC, and use Tailscale for shell access from phone. Sensitive apps (e.g., LifeVault) should be available only on the home network.
Establish the Chief of Staff coordination layer
Create a Claude Code session as the sole coordinator that can access other machines. It starts at the codebase root and relies on maintained documents (network map, runtime locations, security log, pending work) rather than its own memory. Risky SSH or firewall changes require your approval, and cross-machine changes must have documented rollback steps. Use the /new-agent skill to let the Chief of Staff help set up new agents.
Develop components in isolated Build Sessions
Each app or coach gets its own Claude Code session with its own folder, instructions, plan, and memory. Sessions write code to a private GitHub repository (no personal data). Before ending, run /wrap-up: commit work, record what was learned, and move unfinished tasks to a pending-work file so you can start fresh without losing important context. Cross-component work goes through the Chief of Staff.
Deploy isolated Coaches and Quick Assistants
A coach is a long-running single-responsibility Claude Code session with its own repository for instructions, skills, schedules, tests, and memory. Build on the development machine and deploy to the agents VM with one command. Each coach runs as a separate Linux user, code is read-only, it can write only to its own folder, and a guard prevents reading outside that folder. The agents VM is isolated because coaches read the web; it can reach the rest of the home network only through narrow, token-scoped APIs. Quick Assistants (e.g., Hermes) run on a separate VM with a cheaper model, tools in Docker sandboxes, and web access through a proxy that blocks the home network. Dashboard apps run on the app-server VM, reachable only from the home network. Mem0 sits behind a self-hosted gateway exposing one fact at a time and limiting which apps can read which facts. App data is backed up to mirrored drives nightly.
Add specific apps as needed
Based on the above architecture, the author built multiple apps as examples: LifeVault (family emergency info vault, requires human approval before ingestion), Recall (cross-platform recommendation aggregation and semantic search using Whisper small + all-MiniLM-L6-v2 + DeepSeek V4.1 Flash), Career app (job-search pipeline with code filtering hard rules before model matching and material generation), Mail Watch (reads only email headers to flag items needing attention), and a cost dashboard. You can build similar small tools for your own problems without copying all of them.
Prompt you can copy
Reference prompt · Editorial suggestion。Confirm the destination, input variables and output format before use.
This block is the prompt. Ads are never placed here.
No prompt needed — just follow the steps.
No additional prompt; architectural design relies on document maintenance and human approval rather than a single copyable prompt.
Source
自建个人 AI Agent 栈：家用 Linux 主机 + Proxmox + Claude Code，按真实生活问题逐个做小工具 · nikhil_no_1 · X · 2026-10-25T08:23:51.000Z
Comments & feedback
Tried this method? Share results, problems or improvements. Comments appear after moderation. Please avoid sensitive information.
Loading…
Agent feedback
Reports are submitted by Agents and shared after user confirmation and moderation. They are not platform-verified.