The 35-Watt Roommate (Part 4): Docker Cleanup and Surgical Data Recovery

Sometimes you just want to clean up a bit and accidentally tear down half the house. That’s what happened to me last week with my home lab. I wanted to free up some space on the Lenovo M920q and ran a routine Docker cleanup: delete a few unused images, remove orphaned volumes.

But when I opened my Gitea instance afterward, the page was blank. No repositories, no users, everything reset to factory defaults. All my code was gone.

The fatal error in the compose file

The answer was in my docker-compose.yml, where I had made a bad architectural mistake during setup. My volume mapping looked like this:

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    volumes:
      - /var/lib/docker/volumes/infra_volume/_data/gitea_data:/data

At first glance it looks like a normal path. The problem is that /var/lib/docker/volumes/ is Docker’s internal storage. I had reached around Docker and pointed a bind mount straight at the raw data of another volume (infra_volume). Docker didn’t see that dependency, so the cleanup job classified the original volume as unused and deleted it.

Surgical recovery from the Proxmox backup

Restoring the entire VM from backup would have taken forever and overwritten the rest of my setup. I only needed one directory out of the backup, so I had to extract the data directly from the Proxmox backup without ever booting the VM.

The backup lay as a compressed file on my storage:

vzdump-qemu-100-2026_02_23-01_00_02.vma.zst

First, I had to remove the compression:

zstd -d vzdump-qemu-100-2026_02_23-01_00_02.vma.zst

Then I extracted the raw VMA archive using Proxmox’s own tool into a temporary folder:

vma extract vzdump-qemu-100-2026_02_23-01_00_02.vma /mnt/restore

Now I had a raw image file called tmp-disk-drive-scsi0.raw in front of me. You can’t just mount such an image. It contains its own partitions and, in Ubuntu’s case, LVM structures. I had to mount the image as a loop device:

losetup -f
losetup /dev/loop0 /mnt/restore/tmp-disk-drive-scsi0.raw
kpartx -av /dev/loop0

With the kpartx command, I made the partitions in the image visible to the host system. Since Ubuntu uses LVM, I had to wake up the volume group inside the backup:

vgscan
vgchange -ay

The system immediately found the volume group ubuntu-vg. Now I could finally mount the partition like a normal disk and navigate to my lost data:

mount -t ext4 /dev/ubuntu-vg/ubuntu-lv /mnt/vm
cd /mnt/vm/var/lib/docker/volumes/infra_volume/_data/gitea_data

There they were again. My repositories and the Gitea database were intact.

The NFS trap and the SQLite repair

Copying the data back brought the next hurdle. My new destination was an NFS share on my Synology NAS. NFS uses root_squash by default, so the root user of the Proxmox server can’t push files with full permissions to the share, and a normal copy failed.

The solution was rsync without taking over permissions:

rsync -rlt --no-perms --no-owner --no-group /mnt/vm/var/... /mnt/gitea/

The relief was short-lived. The file gitea.db was corrupted, probably because the backup ran at the exact moment the database was being written to. SQLite has a repair mode for this:

sqlite3 gitea.db ".recover" | sqlite3 gitea_fixed.db

If that aborts, you can force SQLite to ignore errors during import:

sqlite3 gitea.db ".recover" | sqlite3 -init <(echo ".bail off") gitea_fixed.db

Two things I’d do differently next time. I ran .recover directly on the file sitting on the NFS mount, and SQLite’s locking doesn’t get along well with NFS, so that’s a second corruption waiting to happen. The safer order is: copy the file to local disk, check how bad it actually is, repair it there, then rsync the fixed copy back.

sqlite3 gitea.db "PRAGMA integrity_check;"

integrity_check tells you upfront whether you’re looking at a handful of misplaced pages or a completely shredded file, before you commit to a recovery pass.

The new setup

After replacing the database and cleanly unmounting everything (umount, vgchange -an, losetup -d), I rebuilt the setup to avoid the mistake.

The new compose file keeps the data on the NAS via a direct mount point instead of a path inside Docker’s internal storage. Whatever Docker does on the Lenovo, the data stays on the NAS:

version: "3.9"

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    restart: always
    networks: [cloudflare]
    ports:
      - "3000:3000"
      - "2222:2222"
    volumes:
      - /mnt/gitea:/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3

  runner:
    image: gitea/act_runner:latest
    container_name: gitea-runner
    restart: always
    depends_on: [gitea]
    networks: [cloudflare]
    environment:
      GITEA_RUNNER_NAME: CI-Runner
      CONFIG_FILE: /data/config.yaml
      GITEA_INSTANCE_URL: http://gitea:3000
      GITEA_RUNNER_REGISTRATION_TOKEN_FILE: /run/secrets/gitea_runner_token
    volumes:
      - runner_data:/data
      - /var/run/docker.sock:/var/run/docker.sock

networks:
  cloudflare:
    external: true

volumes:
  runner_data:
    external: true

secrets:
  gitea_runner_token:
    file: /data/compose/secrets/gitea/gitea_runner_token

What I took away from it

Never use /var/lib/docker/volumes/ or any of Docker’s internal paths as the source for a bind mount. Use --no-perms --no-owner --no-group when rsyncing onto an NFS mount. Learn SQLite’s .recover mode before you need it, but run integrity_check first and repair on local disk, not over NFS. Add a healthcheck so the next failure shows up as a failing container instead of an empty page. And a backup is only worth something once you have actually pulled a single directory out of one and put it back.