🏠 Proxmox Home Server Documentation

Welcome to the official documentation for the 816931.xyz server infrastructure. This guide covers the hybrid cloud/home architecture, networking protocols, service maps, and maintenance procedures.

🏗️ System Architecture

Our infrastructure utilizes a Hybrid Gateway Model, leveraging a public Oracle VPS as a secure entry point to a local Proxmox Virtual Environment (PVE) at https://192.168.0.149:8006/.

1. The Gateway (Oracle VPS)

Public IP: 161.118.178.176
SSH User: ubuntu (Shortcut: ssh vps)
Role: Acts as a reverse proxy and secure entrypoint.
Mechanism: Caddy (Unified Infrastructure).
- Local Services: status.816931.xyz, syncthing.816931.xyz, and vpn.816931.xyz (wg-easy) are terminated locally with automatic HTTPS.
- Home Services: All other *.816931.xyz traffic is reverse proxied over WireGuard to http://10.10.10.2:80 (LXC 109 NATs 80/443 to home Caddy at 192.168.0.129).
- Stealth Services: Certain hostnames like sta.816931.xyz are proxied without Homepage/Gatus visibility. The target LXC 119 is currently stopped; the route is left deliberately.
- Automation: Configuration is automatically generated via Ansible (deploy_vps.yml / deploy_vps_caddy.yml).

2. The Internal Host (Proxmox)

Internal IP: 192.168.0.149
SSH User: root (Shortcut: ssh proxmox)
Web UI: https://192.168.0.149:8006/
Also on host: VirtualHere USB (vhusbd on :7575), Filebrowser on :8080 (LAN-only, not Ansible-managed).

3. The Bridge (WireGuard)

Primary Tunnel (VPN): WireGuard connects VPS (10.10.10.1) to Home LXC 109 (10.10.10.2). Source of truth: wireguard_tunnel_ip in infrastructure_vars.yml.
NAT: LXC 109 forwards ports 80/443 to home Caddy (192.168.0.129).
MTU: Keep WireGuard MTU at 1280 to avoid SSL handshake failures over the tunnel.
Deprecated: wud-tunnel.service on the Proxmox host is disabled and dead. WUD was replaced by Gatus; do not restart it for tunnel recovery.

4. Home Traffic Controller (Caddy)

LXC ID: 106 (192.168.0.129)
Role: Handles SSL (Let's Encrypt DNS-01 via Cloudflare) and internal routing for all Proxmox services.
Automation: Configuration is automatically generated via Ansible.

5. Infrastructure Automation (Ansible)

LXC ID: 110 (Management node)
Tool: Ansible
Role: Orchestrates the configuration of Caddy (Home & VPS), Gatus, Homepage, and CoreDNS from a single source of truth.
Source of Truth: /root/homeserver/ansible/inventory/infrastructure_vars.yml
Primary Playbooks:
- deploy_infra.yml: Deploys Home Caddy, Homepage, and CoreDNS.
- deploy_vps.yml / deploy_vps_caddy.yml: Deploys VPS Caddy Gateway.
- deploy_all_services.yml: Master orchestrator for home Docker services.
- deploy_jiotv.yml / deploy_tataplay.yml: Per-app deploy (JioTV = official GHCR image + nginx edge; TataPlay = vendored PHP app).
- deploy_backups.yml: Intended to manage Proxmox backup jobs — no jobs are active today (/var/lib/vz/dump empty, /etc/pve/jobs.cfg has none). Treat destroyed LXCs as unrecoverable until fixed.
Advanced Features: Supports Multi-Container Sidecars and Ansible Vault for encrypted secrets (secrets.yml).
Also on 110 (not Ansible-managed): Semaphore UI + Tailscale (100.70.140.33).

6. Backup Engine (Syncthing)

Architecture: Uses a "3-Way Bridge" (iPad/PC ↔ VPS Relay ↔ Home HDD).
Relay (VPS): syncthing.816931.xyz (Port 8384).
Home Target: synchome.816931.xyz (LXC 108, Port 8086).
Storage: Files are synced to /mnt/external_drive on Proxmox.

🌐 Networking & DNS

Split DNS (CoreDNS)

LXC ID: 117 (192.168.0.14)
Role: Sole internal DNS authority (Technitium removed).
Function:
- Internal: Resolves domain names to the local Caddy IP (192.168.0.129) for Gigabit speeds.
- Automation: Zone file db.816931.xyz is automatically generated via Ansible based on the global infrastructure variables.
- External: Cloudflare uses a Wildcard A Record (*) pointing all subdomains to the Oracle VPS public IP (161.118.178.176).
Cloudflare Credentials:
- Zone ID: 4046ccfee8a888ec734fb698534aab88
- API Token: <REDACTED_CLOUDFLARE_TOKEN> (DNS Edit permissions)

🚀 Service Inventory

Cloud Hosted (VPS)

Service	URL	Role
Gatus	`https://status.816931.xyz`	Monitoring as Code status page.
Syncthing Relay	`https://syncthing.816931.xyz`	24/7 Always-on sync bridge.
wg-easy	`https://vpn.816931.xyz`	Personal VPN (CGNAT escape hatch); UDP 51820.

Home Hosted (Proxmox / Docker 108 unless noted)

Service	Access Type	Notes
Homepage	`https://homepage.816931.xyz`	Dashboard (Docker 108, `:8082`).
Syncthing Home	`https://synchome.816931.xyz`	Backup target → `/mnt/external_drive` (Docker 108, `:8086`).
Dispatcharr	`https://dispatcharr.816931.xyz`	IPTV manager; EPG config lives in its DB, not Ansible (Docker 108, `:9191`).
JioTV Go	`https://jiotv.816931.xyz`	Official GHCR image + nginx edge; EPG via Dispatcharr + mitthu786 (Docker 108, `:5002`).
TataPlay	`https://tataplay.816931.xyz`	Widevine PHP app → OwnTV / Widevine clients — not Dispatcharr (Docker 108, `:8095`).
Stremio	`https://stremio.816931.xyz`	Backend `:11470` + nginx TV GUI on `:8080` (Docker 108).
Erugo	`https://share.816931.xyz`	File sharing (Docker 108, `:8089`).
Spliit	`https://spliit.816931.xyz`	Expense splitting + postgres sidecar (Docker 108, `:3005`).
Suwayomi	`https://suwayomi.816931.xyz`	Manga server (Docker 108, `:8084`).
ConvertX	`https://convertx.816931.xyz`	File conversion (Docker 108, `:8085`).
BentoPDF	`https://bentopdf.816931.xyz`	PDF tools (Docker 108, `:3000`).
Paperless-ngx	`https://paperless.816931.xyz`	Document management + redis (Docker 108, `:8000`).
Honey	`https://honey.816931.xyz`	Static nginx (Docker 108, `:8088`).
Documentation	`https://docs.816931.xyz`	MkDocs site (Docker 108, `:8008`).
Grafana / Loki / Prometheus	`grafana.` / `loki.` / `prometheus.`	Observability stack (Docker 108).
Home Assistant	`https://ha.816931.xyz`	VM 100 (`192.168.0.228:8123`).
VirtualHere	Proxmox `:7575`	USB over IP (`vhusbd` unit).
`sta.` (stealth)	`https://sta.816931.xyz`	Route still in vars → `192.168.0.100` (LXC 119); container stopped, excluded from Gatus. Left as-is deliberately.
Headwind MDM	`https://mdm.816931.xyz`	Declared but not running (Caddy/cert exist; no container on 108 as of 2026-08-02).

🔄 Monitoring & Updates

We use Gatus for "Monitoring as Code" and Centralized Update Management via Ansible.

Gatus (Monitoring as Code)

Gatus is the production monitoring stack (Uptime Kuma / AutoKuma / WUD are gone). - Location: VPS Docker, port 8085 (proxied via VPS Caddy). - Config Path: /home/ubuntu/gatus/config/config.yaml on the VPS — never edit the Gatus SQLite DB by hand. - Automation: Endpoints are generated from infrastructure_services (+ manual_gatus_endpoints) in infrastructure_vars.yml. Use gatus_monitor: false to opt out. - SSL: VPS Caddy terminates TLS for status.816931.xyz.

Homepage (Automation)

Homepage provides a centralized dashboard on LXC 108. - Automation: services.yaml is automatically generated via Ansible. - Clean State: Any service removed from infrastructure_vars.yml is automatically removed from the dashboard upon the next Ansible run, ensuring no residual entries. - Grouping: Services are automatically categorized into Core Services, Media, Tools, and Infrastructure based on the infrastructure_vars.yml configuration. - Duplicate Prevention: When using Ansible automation, Service Discovery should be disabled in settings.yaml to prevent Docker labels from creating redundant entries. - 🎨 Hybrid Icon Strategy: - The homepage_services.yaml.j2 template uses extension-based detection. - Local Icons: If the icon field ends in .png or .svg, it is served locally from /root/services/homepage/config/icons/ (mapped to /icons/filename.png). - Remote Slugs: If no extension is present, it defaults to the Homarr Dashboard Icons CDN. - Notable Local Icons: - jio.png (JioTV Go) - honey.png (Honey) - picoshare.png (Used for Erugo) - mkdocs.png (Documentation) - Slug Discovery: Search the Homarr metadata.json for slugs. Use curl -I to verify jsDelivr resolution before committing to a remote slug.

Container Organization

All containers on LXC 108 are organized into individual folders: * Path: /root/services/<service_name>/docker-compose.yml * Volumes: Persistent data is mapped to ./data/ relative to the compose file.

JioTV Go (Official Image)

JioTV Go runs from the official GHCR image, managed by playbooks/deploy_jiotv.yml (kept separate from the generic compose loop because of the nginx edge sidecar): * Image: ghcr.io/jiotv-go/jiotv_go:latest — GHCR does not publish semver tags (e.g. v3.18.1 → manifest unknown); :latest tracks releases. Upstream: https://github.com/JioTV-Go/jiotv_go. * Edge: jiotv-edge (nginx:1.27-alpine) publishes host port 5002, serves the webOS TV UI from ./webos, and proxies API/player to jiotv-go:5001. * Credentials Persistence: ./data:/app/.jiotv_go volume — the Jio login (OTP) survives container recreates. If streams start 403ing with "Unable to refresh tokens" in the logs, re-login at https://jiotv.816931.xyz. * Updates: Pulled nightly by maintenance.yml (same path as other Docker services). No source clone, no /etc/cron.d/jiotv-go-update. * History: Until mid-2026 this was a local source build (and briefly tracked a rewritten fork). Switched to GHCR once official images were published and matched release v3.18.1.

JioTV EPG (Program Guide)

JioTV Go cannot supply a program guide — Jio retired their EPG API (jiotv.data.cdn.jio.com/apis/v1.3/getepg/ now returns HTTP 404), so jiotv_go epg gen produces channel definitions with zero <programme> entries. Do not rely on http://<jiotv>:5002/epg.xml.gz for guide data. * Working source: mitthu786/tvepg, an actively-maintained Indian EPG that uses JioTV's numeric channel IDs (e.g. 144 = Colors HD), so it maps to our channels by tvg-id with no name-matching. URL (jsDelivr CDN, gzip): https://cdn.jsdelivr.net/gh/mitthu786/tvepg/jiotv/epg.xml.gz. Covers ~today ±1 day, updated daily. * Avoid: fhux.short.gy/epg.xml.gz (→ gitlab.com/anbuchelva/epg) — recommended in Telegram groups but abandoned since Dec 2025; its data is 7+ months stale and clients silently discard it as past-dated. * Where it's configured: Dispatcharr EPG source "JioTV EPG (mitthu786)" (LXC 108, dispatcharr.816931.xyz), 24h auto-refresh. This lives in Dispatcharr's DB, not in Ansible — restore it manually via the web UI (Settings → EPG) or the API if the container is rebuilt from scratch. Consumed by Kodi/LibreELEC (PVR IPTV Simple) and any client, via Dispatcharr's /output/epg. * Delivery note: Dispatcharr's default stream profile was changed from ffmpeg to Redirect (default_stream_profile in stream_settings) — the ffmpeg remux stalled JioTV's token-rotating HLS and caused choppy playback. Redirect makes clients play JioTV Go's HLS directly (JioTV Go handles Jio token refresh); Dispatcharr stays out of the video data path.

TataPlay (Widevine → OwnTV)

Tata Play live channels are Widevine DRM (not ClearKey). They therefore need a client with a Widevine CDM — OwnTV / OTT Navigator / TiviMate / Kodi (Android's built-in CDM). They cannot go through Dispatcharr/dispatchwrapparr, which is ClearKey-only. (This was learned the hard way — see the note at the end.)

App: DenverIsAlive TP_v1.5 (plugin 8183) — a PHP app that turns a watch.tataplay.com account into a Widevine M3U. Vendored in-repo at ansible/templates/tataplay-app/ (it ships as a zip, no upstream git). Runs as php:8.3-apache on Docker 108, host port 8095, tataplay.816931.xyz. Managed by playbooks/deploy_tataplay.yml (route-only entry in infrastructure_vars.yml).
How it works: login.php does the OTP login → secure/_sessionData (accessToken/refreshToken). Playlist.php emits an M3U with #KODIPROP:inputstream.adaptive.license_type=com.widevine.alpha and a jwt.php?id=<id> license URL. hmac.php pulls the authed channel details (tm.tapi.videoready.tv), decrypts the AES-encrypted stream URL, and captures the Akamai hdntl token from a Set-Cookie (using UA Shraddha/5.0); play.php proxies the manifest with that token on the segments; jwt.php/widevine.php mint a JWT and 307-redirect the player's Widevine challenge to Irdeto. The channel catalog is the static secure/TP_Custom.json.
Persistence: bind-mounted volumes ./secure (session + catalog) and ./cache (token cache), both owned by container www-data (uid 33). secure/ is web-denied via its own .htaccess.

One-time setup: 1. Deploy: ansible-playbook … playbooks/deploy_tataplay.yml. For a new route-only host also run deploy_proxy.yml (home Caddy/cert), deploy_coredns.yml (internal DNS → home proxy) and deploy_vps.yml (VPS Caddy block → proxies the host through the WireGuard tunnel; its own HTTP-01 cert). All public *.816931.xyz resolve to the VPS, so without the VPS block a DoH browser gets SSL_ERROR_INTERNAL_ERROR_ALERT even when the home cert is fine. 2. Log in once at https://tataplay.816931.xyz/login.php (Indian mobile + OTP) → writes secure/_sessionData. 3. Add M3U https://tataplay.816931.xyz/Playlist.php to OwnTV (or any Widevine player) — M3U type, auto-refresh off. Full lineup plays with 8-day catchup.

Why not Dispatcharr: the original attempt used drmlive/tataplay (which mislabels the DRM as clearkey) + the dispatchwrapparr plugin. Tata Play is Widevine, so its tp.drmlive-01.workers.dev "clearkey" worker returns null and channels play blank; dispatchwrapparr has no Widevine CDM and can never decrypt it. CustTermux "works" only because it runs the same PHP tool on Android and plays through an ExoPlayer with Android's Widevine CDM — the exact model as our container + OwnTV.

Caddy TLS: Cloudflare DNS-01 gotchas (fixed 2026-07-23)

Adding tataplay surfaced two independent breakages in Caddy's Cloudflare DNS-01 cert issuance that had already killed mdm/jiotv renewals (every new/renewing cert failed; existing cached certs kept serving):

New Cloudflare token format. Cloudflare now issues prefixed API tokens (cfut_/cfat_). The old caddy-dns/cloudflare module (v0.2.2) rejected them at startup with "API token appears invalid" (fixed in PR #123, released in v0.2.4). Caddy was rebuilt with the fixed module and the binary swapped on the proxy (old one backed up as /usr/bin/caddy.*~). Rebuild command (run in a Docker caddy:*-builder):
```
xcaddy build v2.10.2 --with github.com/caddy-dns/cloudflare@v0.2.4
```
> The Caddy binary is not yet Ansible-managed — only caddy.service + Caddyfile are. If Caddy is reinstalled/upgraded, rebuild with the module ≥ v0.2.4.
Split-horizon DNS shadowing the challenge. The proxy's primary resolver is the internal CoreDNS (192.168.0.14), which is authoritative for 816931.xyz and returns "no record" for the _acme-challenge TXT — so Caddy's propagation self-check never saw the record and timed out. Fixed in Caddyfile.j2 by pinning the challenge to public resolvers via a per-site tls { dns cloudflare …; resolvers 1.1.1.1 8.8.8.8 } snippet (cf_tls), replacing the global acme_dns. IP-only sites (VirtualHere) skip it and use Caddy's internal CA.

The Cloudflare token lives in the vault (vault_cloudflare_api_token); if it's rotated again, caddy validate will fail fast on the next deploy_proxy.yml run.

Interim access if the cert is ever down again: HSTS (includeSubDomains) forces HTTPS on every *.816931.xyz host, so the raw IP http://192.168.0.195:8095/ is the fallback for the TataPlay login. The DRM pipeline itself runs over HTTP on the media-network (http://tataplay/…), independent of the cert.

Caveats: depends on the third-party tp.drmlive-01.workers.dev / tp.secure-kid.workers.dev Cloudflare workers for key extraction (breaks if they go down); needs an active Tata Play subscription + a periodically-refreshed OTP session. The Akamai segment CDN geo-gates on X-Forwarded-For — the adapters send an Indian IP.

How to Update

Updates are automated via a systemd timer on the Management Node (LXC 110). - Manual Trigger: cd /root/homeserver/ansible && ansible-playbook -i inventory/hosts.ini playbooks/maintenance.yml --vault-password-file /root/homeserver/.vault_pass - Schedule: Daily at 03:00 AM. - Log Location: journalctl -u ansible-maintenance on LXC 110.

🛠️ Maintenance & Troubleshooting

Self-Healing Protocols

WireGuard Watchdog (LXC 109): Cron job /usr/local/bin/wg-watchdog.sh pings 10.10.10.1 every 2 minutes. If it fails, it restarts wg-quick@wg0.
Systemd Resiliency: Restart=always is enabled for critical services with a 10s delay.
Boot recovery: CoreDNS (LXC 117) and the Management Node (LXC 110) must have onboot: 1 so DNS and Ansible survive a Proxmox reboot.
Not self-healing: wud-tunnel.service is disabled/dead — ignore any old runbooks that say to restart it.

Common Connectivity Fixes

Tunnel is down: On LXC 109, check/restart WireGuard (systemctl restart wg-quick@wg0) or wait for the watchdog. Confirm with a ping to 10.10.10.1. Do not restart wud-tunnel on Proxmox.
SSH Public Key Denied: If the private key is encrypted with a passphrase, the CLI agent cannot connect non-interactively. Use ssh-add or remove the passphrase (ssh-keygen -p) to allow automation.
DNS Propagation Delays: When adding new subdomains, Gatus may show them as Down due to stale resolver cache. On the VPS: sudo resolvectl flush-caches. As a fallback, add the record to the VPS /etc/hosts.
New public hostname SSL errors: A new *.816931.xyz host needs deploy_proxy.yml + deploy_coredns.yml + deploy_vps.yml. Skipping the VPS block yields SSL_ERROR_INTERNAL_ERROR_ALERT even when the home cert is fine.
Homepage "Host validation failed": Ensure the HOMEPAGE_ALLOWED_HOSTS environment variable is set to your dashboard domain (e.g., homepage.816931.xyz).
Homepage Missing Icons:
1. Check the slug on Homarr / dashboardicons.com.
2. For custom icons, host them in /root/services/homepage/config/icons/ and map to /app/public/icons/.
3. Use the label homepage.icon=/icons/filename.png (explicit path).
4. Important: Homepage discovery automatically adds /icons/ if you only provide a filename. If you provide /icons/file.png, it becomes /icons/file.png internally. Avoid double slashes.
5. Download Note: Always verify curled icons with the file command to ensure they aren't HTML error pages.
LXC Docker not responding: Unprivileged Docker LXCs need nesting=1 and keyctl=1. Verify with pct config 108.

CLI Rules (For AI & Admins)

Service Paths: Always operate within /root/services/<app>/ when editing compose files.
NEVER use the && operator in shell commands; execute them sequentially.
LXC IDs: Always verify actual IPs via pct list before configuring tunnels.
PROXMOX LXC COMMANDS: When executing commands inside an LXC from a remote SSH session, pct exec can misinterpret arguments. Use the -- separator to prevent this and wrap the command in bash -c.
- Correct Example: ssh proxmox 'pct exec 108 -- bash -c "docker ps -a"'

Last Updated: August 5, 2026