first commit
This commit is contained in:
Stanislav Hubacek
547
README.md
Normal file
547
README.md
Normal file
@@ -0,0 +1,547 @@
|
|||||||
|
# Homelab Provisioning Portal — MVP Spec
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
A small internal web application that lets me provision Proxmox LXC containers and selected VMs through a simple web form, then optionally integrate them into my existing homelab workflow:
|
||||||
|
|
||||||
|
- Proxmox provisioning
|
||||||
|
- initial package/profile setup
|
||||||
|
- Git-backed Caddy config generation
|
||||||
|
- optional DNS integration
|
||||||
|
- optional monitoring integration
|
||||||
|
|
||||||
|
The portal is not meant to be a generic public PaaS. It is an internal operator tool for a controlled homelab environment.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Core principles
|
||||||
|
|
||||||
|
### 1. GUI for input, API-driven execution
|
||||||
|
The user experience should be a simple web form, but the backend should orchestrate everything through APIs, scripts, and structured jobs.
|
||||||
|
|
||||||
|
### 2. Profiles over chaos
|
||||||
|
Instead of exposing every package and configuration toggle directly, the app should use a small number of workload profiles.
|
||||||
|
|
||||||
|
### 3. Git as source of truth where possible
|
||||||
|
Reverse proxy configuration should not be edited directly. The portal should write service definitions into the Git repo and let the existing deployment pipeline handle Caddy.
|
||||||
|
|
||||||
|
### 4. Jobs, not blocking requests
|
||||||
|
Provisioning should run as background jobs with visible status, logs, and clear success/failure states.
|
||||||
|
|
||||||
|
### 5. Safe defaults
|
||||||
|
The portal should prefer prevalidated combinations of:
|
||||||
|
- node
|
||||||
|
- storage
|
||||||
|
- template
|
||||||
|
- network mode
|
||||||
|
- package sets
|
||||||
|
- exposure rules
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## MVP scope
|
||||||
|
|
||||||
|
### In scope
|
||||||
|
- create LXC containers
|
||||||
|
- create from a selected profile
|
||||||
|
- set hostname, CPU, RAM, disk, node, storage, network mode
|
||||||
|
- optional static IP or DHCP
|
||||||
|
- optional package installation through profile provisioning
|
||||||
|
- optional generation of Caddy service YAML in Git repo
|
||||||
|
- job history and logs
|
||||||
|
|
||||||
|
### Out of scope for MVP
|
||||||
|
- full VM lifecycle
|
||||||
|
- Terraform integration
|
||||||
|
- approvals/workflows
|
||||||
|
- user accounts / RBAC beyond a single trusted operator
|
||||||
|
- advanced DNS API automation
|
||||||
|
- full secrets management UI
|
||||||
|
- rollback orchestration across all systems
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Profiles for MVP
|
||||||
|
|
||||||
|
### 1. Static Website
|
||||||
|
Purpose:
|
||||||
|
- simple web presentations / landing pages / personal sites
|
||||||
|
|
||||||
|
Provisioning:
|
||||||
|
- Debian LXC
|
||||||
|
- nginx or apache installed
|
||||||
|
- document root prepared
|
||||||
|
- optional domain
|
||||||
|
- optional Caddy service YAML generated
|
||||||
|
|
||||||
|
### 2. Generic Service
|
||||||
|
Purpose:
|
||||||
|
- small internal services
|
||||||
|
|
||||||
|
Provisioning:
|
||||||
|
- Debian LXC
|
||||||
|
- base utilities
|
||||||
|
- optional Docker
|
||||||
|
- optional node exporter
|
||||||
|
- no public exposure by default
|
||||||
|
|
||||||
|
### 3. Monitoring Node
|
||||||
|
Purpose:
|
||||||
|
- internal monitoring or utility workloads
|
||||||
|
|
||||||
|
Provisioning:
|
||||||
|
- Debian LXC
|
||||||
|
- node exporter
|
||||||
|
- curl / git / basic tools
|
||||||
|
- no public exposure by default
|
||||||
|
|
||||||
|
### 4. Web App
|
||||||
|
Purpose:
|
||||||
|
- internal or public HTTP application
|
||||||
|
|
||||||
|
Provisioning:
|
||||||
|
- Debian LXC
|
||||||
|
- web server or runtime preset
|
||||||
|
- optional Caddy service YAML
|
||||||
|
- optional health endpoint setting stored in service definition
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## User flow
|
||||||
|
|
||||||
|
### Step 1 — Choose workload type
|
||||||
|
- LXC
|
||||||
|
- VM (disabled or marked as coming later in MVP)
|
||||||
|
|
||||||
|
### Step 2 — Choose profile
|
||||||
|
- Static Website
|
||||||
|
- Generic Service
|
||||||
|
- Monitoring Node
|
||||||
|
- Web App
|
||||||
|
|
||||||
|
### Step 3 — Basic settings
|
||||||
|
- hostname
|
||||||
|
- description
|
||||||
|
- target node
|
||||||
|
- storage
|
||||||
|
- CPU cores
|
||||||
|
- RAM (MB)
|
||||||
|
- disk size (GB)
|
||||||
|
- start after creation (yes/no)
|
||||||
|
|
||||||
|
### Step 4 — Network
|
||||||
|
- DHCP / Static IP
|
||||||
|
- static IP address (if selected)
|
||||||
|
- gateway (optional)
|
||||||
|
- bridge
|
||||||
|
- VLAN tag (optional)
|
||||||
|
|
||||||
|
### Step 5 — Service integration
|
||||||
|
- domain name (optional)
|
||||||
|
- expose via Caddy (yes/no)
|
||||||
|
- public or internal service
|
||||||
|
- enable monitoring (yes/no)
|
||||||
|
|
||||||
|
### Step 6 — Review
|
||||||
|
Show:
|
||||||
|
- chosen node
|
||||||
|
- chosen profile
|
||||||
|
- resources
|
||||||
|
- network settings
|
||||||
|
- generated service definition preview if relevant
|
||||||
|
- list of post-provision actions
|
||||||
|
|
||||||
|
### Step 7 — Create job
|
||||||
|
- portal stores job
|
||||||
|
- worker executes it
|
||||||
|
- UI shows progress and logs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proposed architecture
|
||||||
|
|
||||||
|
## Frontend
|
||||||
|
Simple server-rendered UI.
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- FastAPI + Jinja templates
|
||||||
|
- minimal JS only where needed
|
||||||
|
|
||||||
|
Why:
|
||||||
|
- lightweight
|
||||||
|
- easy to deploy in LXC
|
||||||
|
- low maintenance
|
||||||
|
- fast iteration
|
||||||
|
|
||||||
|
## Backend
|
||||||
|
FastAPI application with:
|
||||||
|
- form handling
|
||||||
|
- validation
|
||||||
|
- job creation
|
||||||
|
- background execution
|
||||||
|
- job log viewing
|
||||||
|
|
||||||
|
## Persistence
|
||||||
|
SQLite for MVP.
|
||||||
|
|
||||||
|
Tables:
|
||||||
|
- jobs
|
||||||
|
- job_logs
|
||||||
|
- profiles
|
||||||
|
- created_resources (optional for tracking)
|
||||||
|
|
||||||
|
## Execution layer
|
||||||
|
The backend should call a dedicated internal execution layer, not shell out from route handlers directly.
|
||||||
|
|
||||||
|
Modules:
|
||||||
|
- proxmox.py
|
||||||
|
- gitops.py
|
||||||
|
- provision.py
|
||||||
|
- jobs.py
|
||||||
|
- models.py
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Backend modules
|
||||||
|
|
||||||
|
### proxmox.py
|
||||||
|
Responsibilities:
|
||||||
|
- create LXC
|
||||||
|
- optionally start LXC
|
||||||
|
- poll for completion
|
||||||
|
- retrieve created CTID / details
|
||||||
|
|
||||||
|
Possible implementation paths:
|
||||||
|
- Proxmox API directly
|
||||||
|
- wrapper around `pct` CLI if app runs on a trusted node
|
||||||
|
|
||||||
|
Recommendation:
|
||||||
|
- use Proxmox API for long-term cleanliness
|
||||||
|
- CLI wrapper acceptable for MVP if simpler in your environment
|
||||||
|
|
||||||
|
### provision.py
|
||||||
|
Responsibilities:
|
||||||
|
- post-create setup inside container
|
||||||
|
- install profile packages
|
||||||
|
- prepare directories
|
||||||
|
- optionally write default nginx/apache config
|
||||||
|
|
||||||
|
Implementation options:
|
||||||
|
- SSH / pct exec
|
||||||
|
- Ansible later
|
||||||
|
|
||||||
|
Recommendation for MVP:
|
||||||
|
- use `pct exec` from a trusted node if the app runs close to Proxmox
|
||||||
|
- move to Ansible later if desired
|
||||||
|
|
||||||
|
### gitops.py
|
||||||
|
Responsibilities:
|
||||||
|
- write service YAML definitions into repo
|
||||||
|
- validate generated content
|
||||||
|
- git add / commit / push
|
||||||
|
|
||||||
|
Important:
|
||||||
|
- this module should only touch approved repo paths
|
||||||
|
- no arbitrary file writes from user input
|
||||||
|
|
||||||
|
### jobs.py
|
||||||
|
Responsibilities:
|
||||||
|
- create job records
|
||||||
|
- store state transitions
|
||||||
|
- append logs
|
||||||
|
- expose job status to UI
|
||||||
|
|
||||||
|
States:
|
||||||
|
- pending
|
||||||
|
- running
|
||||||
|
- succeeded
|
||||||
|
- failed
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data model (MVP)
|
||||||
|
|
||||||
|
### Job
|
||||||
|
- id
|
||||||
|
- created_at
|
||||||
|
- started_at
|
||||||
|
- finished_at
|
||||||
|
- type
|
||||||
|
- profile
|
||||||
|
- payload_json
|
||||||
|
- status
|
||||||
|
- summary
|
||||||
|
- result_json
|
||||||
|
- error_message
|
||||||
|
|
||||||
|
### JobLog
|
||||||
|
- id
|
||||||
|
- job_id
|
||||||
|
- timestamp
|
||||||
|
- level
|
||||||
|
- message
|
||||||
|
|
||||||
|
### ProfileDefinition
|
||||||
|
Can be hardcoded initially or stored in code/YAML.
|
||||||
|
|
||||||
|
Suggested profile fields:
|
||||||
|
- name
|
||||||
|
- description
|
||||||
|
- supported_kind
|
||||||
|
- default_packages
|
||||||
|
- allow_public_exposure
|
||||||
|
- allow_domain
|
||||||
|
- default_web_server
|
||||||
|
- post_create_actions
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Validation rules
|
||||||
|
|
||||||
|
### General
|
||||||
|
- hostname required, restricted format
|
||||||
|
- RAM / disk / CPU must be from allowed ranges
|
||||||
|
- node must be from approved node list
|
||||||
|
- storage must be from approved storage list
|
||||||
|
- profile must be from approved profile list
|
||||||
|
|
||||||
|
### Network
|
||||||
|
- if static IP selected, IP must be valid
|
||||||
|
- optional uniqueness check against known reserved IPs
|
||||||
|
|
||||||
|
### Domain/Caddy
|
||||||
|
- if expose via Caddy is enabled, domain required
|
||||||
|
- domain uniqueness check against existing service definitions
|
||||||
|
- backend IP generated from created container, not manually trusted from form
|
||||||
|
|
||||||
|
### Safety
|
||||||
|
- never accept arbitrary shell commands from UI
|
||||||
|
- never allow arbitrary package strings in MVP
|
||||||
|
- use whitelisted package sets by profile
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Caddy integration design
|
||||||
|
|
||||||
|
When a workload is marked as web-exposed, the portal should generate a service YAML in the existing homelab repo.
|
||||||
|
|
||||||
|
Example output:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
name: app.example.com
|
||||||
|
type: proxy
|
||||||
|
domain: app.example.com
|
||||||
|
headers: true
|
||||||
|
auth: false
|
||||||
|
backend: 192.168.50.210:80
|
||||||
|
```
|
||||||
|
|
||||||
|
For static website profile with an app server in the container, use proxy mode as well, since the architecture should keep services containerized.
|
||||||
|
|
||||||
|
Portal should then:
|
||||||
|
1. write YAML
|
||||||
|
2. git add / commit / push
|
||||||
|
3. rely on existing Git webhook deployment to update Caddy
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Monitoring integration
|
||||||
|
|
||||||
|
For MVP, monitoring integration can be lightweight.
|
||||||
|
|
||||||
|
Options:
|
||||||
|
- install node exporter in container for selected profiles
|
||||||
|
- add a label/tag to the resource for later tracking
|
||||||
|
- optionally note in job result whether monitoring was enabled
|
||||||
|
|
||||||
|
Prometheus target automation can come later unless you already have a standard discovery mechanism.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## UI pages for MVP
|
||||||
|
|
||||||
|
### 1. Dashboard
|
||||||
|
- recent jobs
|
||||||
|
- quick create button
|
||||||
|
- summary counts
|
||||||
|
|
||||||
|
### 2. New workload form
|
||||||
|
- multi-step or grouped single-page form
|
||||||
|
|
||||||
|
### 3. Job detail page
|
||||||
|
- job status
|
||||||
|
- timestamps
|
||||||
|
- execution logs
|
||||||
|
- generated outputs
|
||||||
|
|
||||||
|
### 4. Profiles page (optional)
|
||||||
|
- read-only overview of supported profiles
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended tech stack
|
||||||
|
|
||||||
|
### App runtime
|
||||||
|
- Python 3
|
||||||
|
- FastAPI
|
||||||
|
- Jinja2
|
||||||
|
- SQLite
|
||||||
|
- Uvicorn
|
||||||
|
|
||||||
|
### Optional frontend enhancement
|
||||||
|
- HTMX for better UX without building a full SPA
|
||||||
|
|
||||||
|
### Deployment target
|
||||||
|
- dedicated LXC container
|
||||||
|
- internal-only exposure via Caddy or VPN
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Security model
|
||||||
|
|
||||||
|
### Access
|
||||||
|
- internal-only
|
||||||
|
- preferably behind VPN or internal-only reverse proxy
|
||||||
|
|
||||||
|
### Secrets
|
||||||
|
Keep these outside Git:
|
||||||
|
- Proxmox API token
|
||||||
|
- Gitea token or SSH deploy key
|
||||||
|
- optional DNS API tokens
|
||||||
|
|
||||||
|
Use environment file or systemd environment variables.
|
||||||
|
|
||||||
|
### Permissions
|
||||||
|
Use a restricted Proxmox API token rather than root password where possible.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What needs to be prepared first
|
||||||
|
|
||||||
|
## A. Proxmox preparation
|
||||||
|
- choose whether MVP uses API or local CLI
|
||||||
|
- prepare at least one approved LXC template
|
||||||
|
- define approved nodes
|
||||||
|
- define approved storage targets
|
||||||
|
- define approved network bridge(s)
|
||||||
|
- define CTID strategy (manual range or automatic lookup)
|
||||||
|
|
||||||
|
## B. Git / Caddy preparation
|
||||||
|
- confirm repo path and branch for service definitions
|
||||||
|
- confirm exact YAML schema used by current generator
|
||||||
|
- define naming convention for generated files
|
||||||
|
- define whether portal commits directly to main or a staging branch
|
||||||
|
|
||||||
|
## C. Host/container provisioning preparation
|
||||||
|
- decide which base OS template to use for LXC
|
||||||
|
- define profile package lists
|
||||||
|
- define default web server choice (nginx or apache)
|
||||||
|
- decide whether provisioning happens through `pct exec` or SSH
|
||||||
|
|
||||||
|
## D. App hosting preparation
|
||||||
|
- create dedicated LXC for the portal
|
||||||
|
- internal DNS name, e.g. `portal.hubacek.cloud`
|
||||||
|
- internal-only reverse proxy rule
|
||||||
|
- prepare environment file for secrets
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Concrete preparation checklist
|
||||||
|
|
||||||
|
### 1. Decide execution method
|
||||||
|
Choose one:
|
||||||
|
- Proxmox API only
|
||||||
|
- Proxmox API for create + `pct exec` for post-provision
|
||||||
|
- local CLI wrapper (`pct`, `qm`) if app runs on a Proxmox node
|
||||||
|
|
||||||
|
Recommended MVP:
|
||||||
|
- Proxmox API for create
|
||||||
|
- `pct exec` for post-provision if app runs on a trusted node
|
||||||
|
|
||||||
|
### 2. Define profile presets
|
||||||
|
Prepare final values for:
|
||||||
|
- Static Website
|
||||||
|
- Generic Service
|
||||||
|
- Monitoring Node
|
||||||
|
- Web App
|
||||||
|
|
||||||
|
For each profile define:
|
||||||
|
- template
|
||||||
|
- packages
|
||||||
|
- default ports
|
||||||
|
- whether Caddy integration is allowed
|
||||||
|
- whether monitoring is installed
|
||||||
|
|
||||||
|
### 3. Define allowed infrastructure values
|
||||||
|
Prepare lists for:
|
||||||
|
- nodes
|
||||||
|
- storage
|
||||||
|
- bridges
|
||||||
|
- VLAN choices if any
|
||||||
|
- RAM options
|
||||||
|
- disk options
|
||||||
|
- CPU options
|
||||||
|
|
||||||
|
### 4. Prepare secrets
|
||||||
|
Need:
|
||||||
|
- Proxmox API token
|
||||||
|
- Gitea credential/token or deploy key
|
||||||
|
|
||||||
|
### 5. Decide repo write behavior
|
||||||
|
Options:
|
||||||
|
- portal commits directly to main
|
||||||
|
- portal writes a branch and you merge manually
|
||||||
|
|
||||||
|
Recommended MVP:
|
||||||
|
- direct commit to main if only you use it
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proposed MVP implementation order
|
||||||
|
|
||||||
|
### Phase 1
|
||||||
|
- app skeleton
|
||||||
|
- SQLite job model
|
||||||
|
- create LXC with Generic Service profile
|
||||||
|
- show job logs
|
||||||
|
|
||||||
|
### Phase 2
|
||||||
|
- add Static Website and Web App profiles
|
||||||
|
- add package installation / post-provision steps
|
||||||
|
- add Caddy Git integration
|
||||||
|
|
||||||
|
### Phase 3
|
||||||
|
- add Monitoring profile
|
||||||
|
- add better job output and validation
|
||||||
|
- add optional DNS integration
|
||||||
|
|
||||||
|
### Phase 4
|
||||||
|
- add VM workflow
|
||||||
|
- add cloud-init templates
|
||||||
|
- add role-based access or approvals if ever needed
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Immediate next actions
|
||||||
|
|
||||||
|
1. Finalize execution strategy
|
||||||
|
2. Finalize profile definitions
|
||||||
|
3. Finalize allowed infrastructure options
|
||||||
|
4. Create dedicated portal container
|
||||||
|
5. Add Proxmox and Gitea credentials outside Git
|
||||||
|
6. Build MVP backend skeleton
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended execution strategy for your homelab
|
||||||
|
|
||||||
|
For your current setup, the most practical MVP is:
|
||||||
|
|
||||||
|
- portal app in dedicated LXC
|
||||||
|
- internal-only exposure
|
||||||
|
- Proxmox API for provisioning
|
||||||
|
- Git repo write for Caddy service definitions
|
||||||
|
- existing webhook pipeline deploys proxy config
|
||||||
|
- post-provision through trusted remote execution later, if not in MVP day one
|
||||||
|
|
||||||
|
This keeps the system aligned with what you already have working instead of replacing it.
|
||||||
|
|
||||||
Reference in New Issue
Block a user