homelab

The 35-Watt Roommate (Part 5): Monitoring, or 'Why is the internet constantly asking for my .php files?'

#homelab#docker#monitoring#grafana#traefik#crowdsec

What, you don’t have your own Grafana dashboard yet? You think that’s just for enterprise environments and way too much hassle for a private homelab?

I used to think the same. When you hear terms like Prometheus, Exporter, Loki, and Promtail, it sounds like massive overhead and a steep learning curve. But I started playing around with it “just for fun,” and I have to say: thanks to Docker, the setup is up and running faster than you can spell “monitoring.”

In this post, I’ll show you how I monitor my setup and how I turn away nasty bots right at the front door.

What do I actually want to know?

You might remember my setup from the article about Cloudflare Tunnels and Docker Stacks. I want transparency for my Nginx instances, the Proxmox host, the Ubuntu server (including SSH logs), and, of course, my Docker containers.

That’s quite a few endpoints, but luckily there’s a matching exporter for almost everything to serve up the data on a silver platter:

  • pve-exporter: Fetches metrics directly from the Proxmox API.
  • node-exporter: The classic for Linux host metrics (CPU, RAM, Disk).
  • cAdvisor: Peeks deep into Docker containers and cgroups.
  • promtail: The “log vacuum” that sends everything to Loki.

The backend consists of Loki (for logs) and Prometheus (for time-series metrics). Architecturally, it looks like this:

Under Docker, the configuration is usually just a copy-paste job. Only the Proxmox exporter requires a bit of manual effort, as you need to create an API token with the right permissions in the PVE interface.

Dashboards: Eye Candy with a Purpose

There’s no need to reinvent the wheel. At grafana.com/dashboards, there’s a massive selection of community dashboards you can import by ID. Here’s my favorites list:

I solved this using Infrastructure as Code (IaC). Provisioning is handled via YAML files so that everything is back up instantly in case of a fresh install. You can find the details in my GitHub repo.

First Discoveries: “Hello, Botnet!”

As soon as the first Nginx logs appeared in Grafana, I was stunned: “Wow, there’s actually a lot of traffic!” A second look quickly revealed: those aren’t fans; those are bots.

I was getting hammered with 404 errors for paths like /tool.php, /admin-footer.php, or /abc.php. Since my sites are based on Astro (static HTML), there is absolutely nothing to gain here for PHP exploits, but the bots keep trying every second anyway.

Log Excerpt (Anonymized): request for /tool.php with HTTP status: 404 from 172.22.0.8 located in IT request for /jga.php with HTTP status: 404 from 172.22.0.8 located in IT

The Solution: Traefik + CrowdSec

At first, I thought Cloudflare’s free plan would handle everything. No such luck. Many of these scans still get through. Since Traefik sits behind the Cloudflare proxy, you normally only see the Cloudflare IP, so a ban would end up blocking the proxy itself.

The solution: CrowdSec, the modern successor to fail2ban, combined with a Traefik plugin. The trick is the label forwardedHeadersCustomName=CF-Connecting-IP. This tells the CrowdSec plugin to pull the real visitor’s IP from the Cloudflare header.

Here is my docker-compose.yml snippet for the dynamic duo:

version: "3.8"

services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: crowdsec
    restart: unless-stopped
    # Entrypoint handles the bouncer key from secrets
    entrypoint: ["/bin/sh", "-c", "export BOUNCER_KEY_traefik=$(cat /run/secrets/crowdsec_bouncer_key) && exec /docker_start.sh"]
    environment:
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/base-http-scenarios"
    volumes:
      - crowdsec-config:/etc/crowdsec
      - crowdsec-data:/var/lib/crowdsec/data
      - traefik-access-logs:/var/log/traefik:ro
    networks:
      - apps
      - monitoring_net
    secrets:
      - crowdsec_bouncer_key

  traefik:
    image: traefik:v3.6.6
    container_name: traefik
    restart: unless-stopped
    depends_on:
      - crowdsec
    command:
      - "--accesslog=true"
      - "--accesslog.filepath=/var/log/traefik/access.log"
      - "--experimental.plugins.crowdsec-bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      - "--experimental.plugins.crowdsec-bouncer.version=v1.4.1"
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.crowdseclapihost=crowdsec:8080"
      - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.forwardedHeadersCustomName=CF-Connecting-IP"
    networks:
      - cloudflare
      - apps

The “Middleware Police” in Action

Now, I just need to add the corresponding middleware label to every web service (like my portfolio or the blog). Every request is checked against the CrowdSec database. If an IP is flagged as “malicious,” Traefik blocks access immediately.

blog-astro-web-prd:
  image: nginx:alpine
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.blog-astro-prd.rule=Host(`www.slashgordon.link`)"
    - "traefik.http.routers.blog-astro-prd.middlewares=crowdsec@docker"

Bottom Line: Is it worth the effort?

Definitely. It’s a reassuring feeling to check Grafana and see the “Banned IPs” count go up while the CPU load of my 35-watt roommate stays stable. Monitoring isn’t just a toy, it’s the immune system of the homelab.

Since activating CrowdSec, the correlation is clearly visible in the dashboard: as soon as a bot starts scanning my .php paths, it gets “sunk” completely after just a few attempts.

Back to all posts