The 35-Watt Roommate (Part 5): Monitoring, or 'Why is the internet constantly asking for my .php files?'

A Grafana dashboard sounds like an enterprise thing, far too much hassle for a private homelab. I used to think so too.

Terms like Prometheus, Exporter, Loki, and Promtail sound like a lot of overhead and a steep learning curve. But I started playing around with it for fun, and thanks to Docker the whole thing was up and running faster than you can spell “monitoring”. This is how I ended up monitoring my setup and turning away bots at the front door.

What do I actually want to know?

You might remember my setup from the article about Cloudflare Tunnels and Docker Stacks. I want visibility into my Nginx instances, the Proxmox host, the Ubuntu server (including SSH logs), and my Docker containers.

That’s a fair number of endpoints, but there’s a matching exporter for almost all of it:

• pve-exporter fetches metrics directly from the Proxmox API.
• node-exporter is the classic for Linux host metrics (CPU, RAM, disk).
• cAdvisor looks inside Docker containers and cgroups.
• promtail is the log vacuum that ships everything to Loki.

The backend is Loki for logs and Prometheus for time-series metrics. It looks like this:

Under Docker the configuration is mostly a copy-paste job. Only the Proxmox exporter needs manual work, since you have to create an API token with the right permissions in the PVE interface.

Dashboards

At grafana.com/dashboards there’s a large selection of community dashboards you can import by ID. My favorites:

• Node Exporter Full: everything about the host.
• Nginx Web Analytics: who’s coming from where?
• cAdvisor Exporter: what are my containers eating?
• SSH Logs: who is trying to log in via SSH?
• CrowdSec Metrics: who’s already been kicked out?

Provisioning runs from YAML files, so everything comes back instantly after a fresh install. The details are in my GitHub repo.

• 

• 

• 

• 

First discoveries: hello, botnet

As soon as the first Nginx logs showed up in Grafana, there was a lot more traffic than I expected. A second look showed that most of it came from bots.

I was getting hammered with 404s for paths like /tool.php, /admin-footer.php, or /abc.php. My sites are static HTML built with Astro, so there is nothing here for a PHP exploit to find, but the bots keep trying every second anyway.

Log excerpt (anonymized): request for /tool.php with HTTP status: 404 from 172.22.0.8 located in IT request for /jga.php with HTTP status: 404 from 172.22.0.8 located in IT

The fix: Traefik and CrowdSec

I thought Cloudflare’s free plan would catch all of this. It doesn’t, plenty of these scans still get through. And since Traefik sits behind the Cloudflare proxy, it normally only sees the Cloudflare IP, so a ban would block the proxy itself.

The answer is CrowdSec, the modern successor to fail2ban, plus a Traefik plugin. The label forwardedHeadersCustomName=CF-Connecting-IP tells the CrowdSec plugin to read the real visitor IP from the Cloudflare header.

Here is my docker-compose.yml snippet for the two of them:

version: "3.8"

services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: crowdsec
    restart: unless-stopped
    # Entrypoint handles the bouncer key from secrets
    entrypoint: ["/bin/sh", "-c", "export BOUNCER_KEY_traefik=$(cat /run/secrets/crowdsec_bouncer_key) && exec /docker_start.sh"]
    environment:
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/base-http-scenarios"
    volumes:
      - crowdsec-config:/etc/crowdsec
      - crowdsec-data:/var/lib/crowdsec/data
      - traefik-access-logs:/var/log/traefik:ro
    networks:
      - apps
      - monitoring_net
    secrets:
      - crowdsec_bouncer_key

  traefik:
    image: traefik:v3.6.6
    container_name: traefik
    restart: unless-stopped
    depends_on:
      - crowdsec
    command:
      - "--accesslog=true"
      - "--accesslog.filepath=/var/log/traefik/access.log"
      - "--experimental.plugins.crowdsec-bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      - "--experimental.plugins.crowdsec-bouncer.version=v1.4.1"
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.crowdseclapihost=crowdsec:8080"
      - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.forwardedHeadersCustomName=CF-Connecting-IP"
    networks:
      - cloudflare
      - apps

The middleware in action

From here I just add the middleware label to every web service (portfolio, blog, and so on). Every request gets checked against the CrowdSec database, and if an IP is flagged as malicious, Traefik blocks it immediately.

blog-astro-web-prd:
  image: nginx:alpine
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.blog-astro-prd.rule=Host(`www.slashgordon.link`)"
    - "traefik.http.routers.blog-astro-prd.middlewares=crowdsec@docker"

Is it worth the effort?

For me, yes. It’s reassuring to open Grafana and watch the banned-IPs count climb while the CPU load of my 35-watt roommate stays flat. The dashboard also makes the effect obvious: as soon as a bot starts scanning my .php paths, it’s blocked after a few attempts.