homelab

The 35-Watt Roommate (Part 4): Docker Cleanup and Surgical Data Recovery

#homelab#docker#proxmox#gitea#backup#disaster-recovery

Sometimes you just want to clean up a bit and accidentally tear down half the house. That’s exactly what happened to me last week with my Home Lab. I wanted to free up some space on the Lenovo M920q and performed a routine Docker cleanup. Delete a few unused images, remove orphaned volumes. Standard stuff.

But when I called up my Gitea instance afterward, I got the shock of my life: the page was blank. No repositories, no users, everything was reset to factory defaults. All my code was gone.

VM Recovery from Proxmox Backup

The Fatal Error in the Compose File

How could that happen? The answer lay in my docker-compose.yml. I had made a massive architectural error during setup. My volume mapping looked like this:

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    volumes:
      - /var/lib/docker/volumes/infra_volume/_data/gitea_data:/data

At first glance, it looks like a normal path. But the problem is that /var/lib/docker/volumes/ is Docker’s internal vault. I had tricked Docker and accessed the raw data of another volume (infra_volume) directly via a bind mount. Since Docker didn’t understand this dependency, the cleanup job classified the original volume as “unused” and deleted it completely.

Surgical Recovery from the Proxmox Backup

Restoring the entire VM from backup would have taken forever and overwritten the rest of my setup. I only wanted to get to this one directory in the backup. So I had to perform open-heart surgery and extract the data directly from the Proxmox backup without ever booting the VM.

The backup lay as a compressed file on my storage:

vzdump-qemu-100-2026_02_23-01_00_02.vma.zst

First, I had to remove the compression:

zstd -d vzdump-qemu-100-2026_02_23-01_00_02.vma.zst

Then I extracted the raw VMA archive using Proxmox’s own tool into a temporary folder:

vma extract vzdump-qemu-100-2026_02_23-01_00_02.vma /mnt/restore

Now I had a raw image file called tmp-disk-drive-scsi0.raw in front of me. You can’t just mount such an image. It contains its own partitions and, in Ubuntu’s case, LVM structures. I had to mount the image as a loop device:

losetup -f
losetup /dev/loop0 /mnt/restore/tmp-disk-drive-scsi0.raw
kpartx -av /dev/loop0

With the kpartx command, I made the partitions in the image visible to the host system. Since Ubuntu uses LVM, I had to wake up the volume group inside the backup:

vgscan
vgchange -ay

The system immediately found the volume group ubuntu-vg. Now I could finally mount the partition like a normal disk and navigate to my lost data:

mount -t ext4 /dev/ubuntu-vg/ubuntu-lv /mnt/vm
cd /mnt/vm/var/lib/docker/volumes/infra_volume/_data/gitea_data

There they were again. My repositories and the Gitea database were saved.

The NFS Trap and a SQLite Miracle

When copying back, the next hurdle came up immediately. My new destination was a secure NFS share on my Synology NAS. NFS uses root_squash by default, which means the root user of the Proxmox server can’t just push files with full permissions to the share. A normal copy failed.

The solution was rsync without taking over permissions:

rsync -rlt --no-perms --no-owner --no-group /mnt/vm/var/... /mnt/gitea/

The relief was great, but short-lived. The file gitea.db was corrupted, probably because the backup was running at exactly the moment when the database was being written to. Fortunately, SQLite has a powerful repair mode:

sqlite3 gitea.db ".recover" | sqlite3 gitea_fixed.db

If that aborts, you can force SQLite to ignore errors during import:

sqlite3 gitea.db ".recover" | sqlite3 -init <(echo ".bail off") gitea_fixed.db

The New and Secure Setup

After replacing the database and cleanly unmounting everything (umount, vgchange -an, losetup -d), it was time to learn from the mistake.

My new compose file avoids dangerous paths and instead stores the data safely on the NAS via a direct mount point. No matter what Docker does on the Lenovo in the future, the data is safe on the storage:

version: "3.9"

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    restart: always
    networks: [cloudflare]
    ports:
      - "3000:3000"
      - "2222:2222"
    volumes:
      - /mnt/gitea:/data

  runner:
    image: gitea/act_runner:latest
    container_name: gitea-runner
    restart: always
    depends_on: [gitea]
    networks: [cloudflare]
    environment:
      GITEA_RUNNER_NAME: CI-Runner
      CONFIG_FILE: /data/config.yaml
      GITEA_INSTANCE_URL: http://gitea:3000
      GITEA_RUNNER_REGISTRATION_TOKEN_FILE: /run/secrets/gitea_runner_token
    volumes:
      - runner_data:/data
      - /var/run/docker.sock:/var/run/docker.sock

networks:
  cloudflare:
    external: true

volumes:
  runner_data:
    external: true

secrets:
  gitea_runner_token:
    file: /data/compose/secrets/gitea/gitea_runner_token

Key Lessons Learned

What did I learn from this? Never use /var/lib/docker/volumes/ as the source for a bind mount. And above all: a backup is only worth something when you know how to surgically extract exactly the data you need in an emergency.

The most important takeaways:

  • Never use Docker’s internal paths for bind mounts
  • Always use --no-perms --no-owner --no-group with NFS mounts
  • Know how to use SQLite’s recovery mode
  • Test backups regularly and document recovery procedures
Back to all posts