Systems engineering | Ongoing practice
Self-hosted Infrastructure & Automation
Designing and operating a personal service platform, with automated delivery, observability and backup workflows.
Architecture | Observability | Backup & recovery
Service responsibilities
Background
This infrastructure supports the photos, media, source control and personal applications that my family and I use. Keeping control of that data led to services running on home equipment and remote servers, with deployment, networking, monitoring and backups around them.
Where those services belong has been a recurring decision. Remote hosts offer hardware redundancy but distance affects access; home equipment brings direct control and responsibility for power, networking and recovery. As the platform grew, the priorities became clearer: control of data and infrastructure, availability, access speed, operational simplicity and fixed cost.
Design
Early on, I placed considerable weight on disk redundancy and ECC memory, then began evaluating what those requirements meant for each workload. Migration to the home host started with repeatable CI builds and gradually extended to personal applications, while monitoring and backups remained remote.
Service placement reflects storage behaviour, network paths, failure impact and maintenance effort. With the platform primarily serving family needs, the design allows for manual recovery and includes independent backups and recovery procedures. Monitoring runs away from the home application host, and backups span regions and providers.
Implementation
Configuration lives in Git, with Gitea Actions handling application builds and deployment. Containers run across hosts connected by private networking. Prometheus and Grafana collect host and service metrics, while external probes exercise public entry points.
Restic keeps versioned copies of application data and configuration. Shell and Python scripts handle certificate distribution, service discovery, recurring checks and backup reports. Configuration, execution logs and acceptance records provide the context for subsequent maintenance.
Challenges
- Separating hardware appeal from workload requirements. I was reluctant to let go of a powerful server that had been difficult to obtain, and repeatedly considered adding an origin in Asia. Comparing the options led to keeping suitable existing resources and reducing migration work, followed by the recognition that those servers offered more performance than the services needed.
- Collecting data that rarely helped with diagnosis. I considered removing the monitoring stack because much of its data went unused. Reviewing the usual workflow showed that diagnosis started with an alert, followed by investigation on the affected host. Collection needed to reflect that process.
- Keeping data and backups understandable through migrations. Data copied before writes stop needs a final synchronisation, and an old deployment can start again. After several moves, backup repositories named after hosts can also cease to describe their contents.
Solutions
Service load, data volume and access paths became the basis for placement decisions, clarifying the roles of home applications, remote media, monitoring and backups. The home host later moved to Debian running directly on the hardware, removing a VM split that those services no longer needed.
The monitoring changes removed central log collection through Loki and Promtail, retained Prometheus and Grafana, and set the collection interval to one minute at that stage. Diagnosis continued to follow requests through DNS, the entry point, the private network and the application, supported by the relevant metrics and alerts.
Migration procedures stop writes before synchronising the final data. SQLite moves combine a write-ahead log checkpoint, integrity checks, file comparisons and public entry point checks, with the old deployment disabled. Backup naming moved away from host identities, using stable dataset identifiers and a destination matrix to detect missing, failed or stale results. Recovery checks restore selected data and inspect files, databases and configuration, with the scope recorded for each check.
Outcomes
The platform now has defined roles for home applications, remote services and independent backups, supported by migration and recovery records. Subsequent moves can check backup coverage by dataset and responsibility, and public entry point checks form part of delivery acceptance.
Successive iterations clarified where services run, reduced monitoring collection and standardised backup dataset identifiers. Completed recovery checks cover selected files, databases and configuration. Those results guide subsequent maintenance alongside the migration records.