My Home Lab Documentation

Documentation for my home lab CI/CD setup

View the Project on GitHub Auzlex/my-home-lab

← Back to Home

Maintenance Log

This page documents issues, problems, and maintenance activities that occur during the operation of the home lab. Each entry includes the date, problem description, diagnosis steps, resolution, and follow-up notes.

August 04, 2026 - VPS Unreachable — Boot Failure into Emergency Mode (VIREN)

Problem Description

The VIREN OVH VPS became completely unreachable — no ping, no SSH — after a period of normal operation. No external symptom beyond the box being “down”. The system was booting into emergency.target rather than completing a normal multi-user boot, which halted before networking or SSH ever started.

Diagnosis Steps

  1. Initial hypothesis: firewall blocking connections — ruled out (no config check confirmed this).

  2. Checked dnf history for a broken/interrupted package transaction — both relevant transactions showed Status: Ok, ruling out package corruption.

  3. Noted a kernel upgrade (7.1.4-104 → 7.1.5-101) had occurred shortly before the outage via a manual dnf update (not the automatic security-only updater) and treated it as the leading suspect.

  4. Rolled back the default boot kernel to 7.1.4-104 via grubby --set-default as a precaution — this did not resolve the issue on its own, indicating the kernel was not the actual cause.

  5. Rescue-mode chroot sessions initially showed “No journal files were found” for the failed boot, which was misleading — persistent journal logging was in fact already configured and working on the real system; the gap was an artifact of chroot not reliably exposing the live journal state, not an actual logging failure.

  6. Obtained a live boot capture via OVH’s KVM/serial console, which showed the boot reaching emergency.target roughly 40 seconds into boot, after a suspicious gap with no output — the actual diagnostic breakthrough.

  7. Cross-referenced /etc/fstab inside a rescue-mode chroot and identified a stale UUID entry for a previously-attached music drive that had since been removed:

UUID=e2ce6651-9cda-4e34-a1b6-175704a7741e /mnt/music ext4 defaults 0 2

Since the device no longer existed and had no nofail option, systemd treated it as a required mount at boot, blocked on it, timed out, and dropped to emergency.target.

Note: Disk device naming (sda vs sdb) was inconsistent across separate OVH rescue-mode boots, requiring lsblk -f to be re-checked each time before mounting.

Resolution

  1. Commented out the stale /mnt/music line in /etc/fstab from within a rescue-mode chroot.

  2. Switched OVH boot mode back to disk boot and rebooted; confirmed via KVM console that the system reached a normal login prompt without hitting emergency mode.

  3. Confirmed external reachability (ping and SSH on the system’s non-default SSH port) after boot.

  4. Re-tested the kernel upgrade path deliberately: set the default kernel back to 7.1.5-101 and rebooted again to confirm it boots cleanly now that the actual root cause was fixed. Confirmed working — the kernel was never the problem.

Follow-up Notes

  • Any future temporary or removable mount added to /etc/fstab must include the nofail option (and optionally x-systemd.device-timeout=) so a missing device cannot block boot into emergency mode again, e.g.: UUID=... /mnt/music ext4 defaults,nofail 0 2
  • Confirmed persistent journald logging (/var/log/journal) is enabled and retains boot history across reboots, useful for any future incident diagnosis.
  • dnf-automatic is configured for security-only updates with apply_updates = yes; it was not responsible for this incident (the kernel upgrade was from a manual dnf update run, not the automatic timer). No change made to this policy.
  • The OVH KVM/serial console should be the first troubleshooting step for any future 'unreachable, boots but no network' scenario — static inspection via rescue-mode chroot alone was insufficient to identify the actual point of failure; live console output was the key diagnostic.

May 08, 2026 - System Stability Confirmed

Problem Description

Monitoring the Ryzen NAS after the Global C-States fix was applied to ensure the freeze/sleep issues were fully resolved.

Diagnosis Steps

N/A

Resolution

After several tests and a long period of uptime, system stability has been confirmed. The C-States fix successfully prevented the system from entering unrecoverable sleep states.

Follow-up Notes

  • Continue normal operations and monitor if any other Ryzen 1st gen quirks arise.

May 05, 2026 - Resolved Global C-States Sleep Issue on Ryzen

Problem Description

The system would occasionally enter a sleep state from which it could not awake, requiring a hard reset. This is a known issue with first-generation Ryzen processors and Linux.

Diagnosis Steps

Investigated system logs leading up to freezes. Identified that the freezes occurred during idle periods, pointing towards power management and C-States.

Resolution

Disabled Global C-States in the BIOS/UEFI settings, which prevents the CPU from entering the low-power sleep states that cause the system to hang.

Follow-up Notes

  • Monitor system stability over the coming weeks.
  • Check if disabling C-States impacts power consumption significantly.

May 04, 2026 - Hardware Upgrade & OMV8 Migration

Problem Description

The NAS (Casper) needed more computing power for handling all tasks including CI/CD and self-hosted services.

Diagnosis Steps

N/A

Resolution

Rebuilt the NAS hardware from an old AMD A10-7700K processor to an AMD Ryzen 1700 (repurposed old hardware) on AM4 using 16GB of DDR4 (previously 8GB of DDR3). Upgraded the OS drive from a 128GB SATA SSD to a 256GB NVMe SSD, and added two 1TB HDDs in a new ZFS pool called ‘casper-buffer’. During the hardware upgrade, upgraded from OMV7 to OMV8. Successfully exported and reimported the ZFS pools, and reassigned/setup the Gitea server. Added a local Gitea runner called ‘casper-runner’ to handle all CI/CD tasks with the new hardware capabilities.

Follow-up Notes

  • Monitor system performance with the new Ryzen hardware.
  • Ensure all Gitea Actions workflows run smoothly on casper-runner.

February 05, 2026 - Pi-hole Lost Internet Due to Missing Default Gateway in NetworkManager

Problem Description

Devices using Pi-hole DNS lost internet connectivity. The Pi-hole host could communicate with local network devices but could not reach external addresses such as 1.1.1.1. Investigation showed the system had a static IPv4 address configured via NetworkManager (nmcli), but no IPv4 gateway was defined. As a result, no default route existed and the system could not reach upstream DNS servers.

Diagnosis Steps

  1. Confirmed Pi-hole host had a valid local IP address (192.168.0.125) on interface eth0.

  2. Tested connectivity to the router: 
    ping 192.168.0.1

    Result: Successful.

  3. Tested external connectivity: 
    ping 1.1.1.1

    Result: Failed.

  4. Inspected routing table: 
    ip route

    Output showed only the local network route:

    192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.125 metric 100

    The expected default route (default via 192.168.0.1) was missing.

  5. Temporarily restored connectivity by adding a default route manually: 
    sudo ip route add default via 192.168.0.1

  6. Identified root cause: the NetworkManager connection used manual IPv4 configuration but did not define an ipv4.gateway value.

  7. Permanently fixed the issue by adding the gateway to the nmcli connection: 
    nmcli connection modify  ipv4.gateway 192.168.0.1
nmcli connection up </code></pre>

Follow-up Notes

  • Ensure all manual IPv4 configurations include ipv4.addresses, ipv4.gateway, and ipv4.method manual.
  • Verify routing after network changes using 'ip route'.
  • Consider documenting nmcli network configuration to avoid missing gateway parameters in the future.

January 25, 2026 - Updated Pi-hole Gravity and Added Meta Quest Blocklists

Problem Description

Performed a Pi-hole gravity update and added DNS blocklists to prevent Meta Quest Pro devices from contacting Meta services, including firmware update and advertising endpoints. The Meta Quest Pro is discontinued, and updates are intentionally blocked at the DNS level.

Diagnosis Steps

  1. Ran a Pi-hole gravity update to refresh all blocklists: 
    pihole -g
  2. Added the Meta Quest firmware update blocklist: 
    https://raw.githubusercontent.com/ibrah3m/pihole-meta-quest-blocklist/main/hosts-firmware
  3. Added the Meta Quest ads blocklist: 
    https://raw.githubusercontent.com/ibrah3m/pihole-meta-quest-blocklist/main/hosts-ads
  4. Confirmed domains were blocked via Pi-hole query logs when the Meta Quest Pro attempted outbound connections.

Follow-up Notes

  • Monitor Pi-hole logs to ensure Meta-related domains remain blocked.
  • Review blocklists periodically for updates or changes.
  • Be aware that blocking firmware endpoints may affect certain online features or services.

January 21, 2026 - Enforced Pi-hole DNS on All Nodes via Device Configuration including IPv6

Problem Description

Ensured ENLIL, LORIC, AUREL, and Casper Raspberry Pi nodes are using the ENLIL Pi-hole as their DNS server directly on the devices rather than relying on the router settings.

Diagnosis Steps

  1. Checked current DNS settings on all nodes: 
    nmcli device show
  2. Applied Pi-hole DNS directly to each connection profile using nmcli: 
    sudo nmcli connection modify connection ipv4.dns "192.168.0.125" ipv4.ignore-auto-dns yes ipv6.ignore-aut-dns yes ipv4.dns "2a00:23c7:593:6501:ba27:ebff:fe0f:e3f2"
  3. Verified on each device to see Pi-hole as primary DNS: 
    nmcli connection show target_name
  4. Confirmed all nodes resolved domains correctly and queries appeared in Pi-hole logs.

Follow-up Notes

  • Monitor Pi-hole query logs to ensure all devices consistently use it for DNS resolution.
  • Document any new connection profiles that may appear to ensure they also point to Pi-hole.
  • Verify Docker containers respect the device DNS settings for consistency.

January 19, 2026 - Configured LORIC and AUREL to Use ENLIL Pi-hole DNS ipv4 only

Problem Description

Changed the DNS resolving on Raspberry Pi nodes to target the ENLIL Pi-hole hosted on 192.168.0.125.

Diagnosis Steps

  1. Verified current network connections: 
    nmcli connection show
  2. Updated LORIC’s Wi-Fi connection to use Pi-hole: 
    sudo nmcli connection modify BT-RJATFG ipv4.dns "192.168.0.125" ipv4.ignore-auto-dns yes && sudo nmcli connection up BT-RJATFG && nmcli connection show BT-RJATFG

  3. Noted that our home network is on BT, which prevents directly overriding DNS for all devices. Devices currently need to be manually pointed to the Pi-hole.

  4. Observed connection names:
    • LORIC: BT-RJATFG
    • AUREL: netplan-wlan0-BT-RJATFG (likely caused by LORIC losing Wi-Fi randomly; exact cause unknown)
  5. Checked Docker containers, which may ignore host DNS configuration: 
    docker run --rm alpine cat /etc/resolv.conf

This ensures that containers respect the Pi-hole DNS.

  1. Updated devices LORIC and AUREL to follow Pi-hole settings.

Follow-up Notes

  • Monitor DNS queries via Pi-hole to confirm proper resolution and fallback behavior.
  • Consider applying similar DNS configuration to Casper node next.
  • Watch for unexpected network drops that may cause new connection profiles like netplan-wlan0-BT-RJATFG.
  • Document Docker container DNS behavior for future reference.

January 18, 2026 - ENLIL Pi-hole Deployment and Initial Blocklist Tuning

Problem Description

Deployment of Pi-hole on ENLIL (Raspberry Pi 1 Model B) as a dedicated network DNS and filtering node.

Diagnosis Steps

Verified Pi-hole operation via web interface and query logs.

Findings:

  • Pi-hole was successfully serving DNS requests.
  • Traffic was being filtered as expected with no immediate false positives observed.

Resolution

  1. Installed and configured Pi-hole on ENLIL.

  2. Added blocklists sourced from Firebog

    • Selected two lists from each category:
      • Suspicious Lists
      • Advertising Lists
      • Tracking and Telemetry Lists
      • Malicious Lists

  3. Ran a manual Gravity update using the Pi-hole web interface.

  4. Added manual blocking for Facebook-related domains.

  5. Set upstream DNS provider to Cloudflare (temporary).

Follow-up Notes

  • Rule of thumb followed: fewer lists is better to reduce false positives.
  • Continue monitoring query logs for breakage or over-blocking.
  • Revisit upstream DNS choice later if needed.
  • Gradually tune blocklists instead of adding more.

January 17, 2026 - /etc/hosts File Resetting After Reboot

Problem Description

The /etc/hosts file was being reset after system reboots, losing custom DNS resolutions including 192.168.0.124 entries.

Diagnosis Steps

Investigated the cause of hosts file resetting:

  1. Checked cloud-init configuration: 
    cat /etc/cloud/cloud.cfg
  2. Examined hosts template file: 
    cat /etc/cloud/templates/hosts.debian.tmpl

Findings: Cloud-init was managing /etc/hosts and overwriting custom entries on each boot.

Resolution

Disabled cloud-init management of /etc/hosts and updated the template:

  1. Edited cloud-init configuration: 
    sudo nano /etc/cloud/cloud.cfg

    Set: manage_etc_hosts: false

  2. Updated hosts template to include DNS resolution: 
    sudo nano /etc/cloud/templates/hosts.debian.tmpl
  3. Rebooted system to verify changes persist: 
    sudo reboot

The /etc/hosts file now retains custom entries across reboots.

Follow-up Notes

  • Double-check that 192.168.0.124 actually resolves after a reboot.
  • Look at the other Pis to make sure they don’t have the same cloud-init issue.
  • Keep a note of this fix for future setups so it doesn’t catch you off guard.
  • Watch for any weird behavior from cloud-init in the coming weeks.

January 17, 2026 - LORIC WiFi Connection Issue & Runner Offline

Problem Description

LORIC (Raspberry Pi 3 B+ orchestration node) did not respond after a reboot. The device was unreachable over the network, and the Gitea runner service on it was offline.

Diagnosis Steps

  1. Connected Ethernet cable directly to LORIC to establish connectivity and investigate: 
    ip a
  2. Examined wireless interface status: 
    ifconfig wlan0

Findings:

  • The Raspberry Pi had forgotten the WiFi SSID and had no active wireless connection.
  • Even after Ethernet connectivity was restored, the Gitea runner service failed to start due to DNS resolution issues.

Resolution

  1. Reconfigured WiFi settings using Raspberry Pi configuration tool: 
    sudo raspi-config
    • Select System Options > Wireless LAN
    • Enter correct SSID and passphrase
    • Save and reboot
  2. Fixed local DNS resolution to allow the runner service to start: 
    sudo nano /etc/hosts

    Added entries for all relevant nodes.

  3. Restarted Gitea runner: 
    sudo systemctl restart gitea-runner
sudo systemctl status gitea-runner

LORIC reconnected to the network and the runner service resumed normal operation.

Follow-up Notes

  • Keep an eye on LORIC’s WiFi over the next few days to make sure it stays stable.
  • Make sure all Pi nodes have persistent DNS entries so the runner can always find what it needs.
  • For critical services, consider keeping a backup Ethernet connection ready.
  • Watch the runner logs for any future failures that might be network-related.