cloud

JVM Monitoring for Denodo on ECS Fargate with JMX Exporter and CloudWatch

Updated May 12, 2026
#aws#ecs-fargate#jmx#cloudwatch#denodo#java#monitoring#prometheus

Today I’ve got a challenge from my day job that I found pretty interesting. And I’m a little proud of the solution, too.

The Problem

When you’re running a Java application like Denodo, at some point the standard ECS metrics just aren’t enough anymore. CPU and memory are nice, but what’s happening inside the JVM? Heap usage, GC behavior, thread counts, Denodo-specific metrics like active connections and waiting requests. For that, you need JMX.

Our setup: Denodo Virtual Dataport runs as a containerized Java process on AWS ECS Fargate. For monitoring we use Grafana Cloud, which connects to AWS via a CloudWatch datasource. So the monitoring solution and the services are physically separated. The plan was simple: push JMX metrics into CloudWatch, Grafana reads them from there.

The obvious solution would be the native jmx section of the CloudWatch Agent. But it doesn’t work here:

  1. The official CW Agent image is based on FROM scratch. No Java, no shell. The jmx section internally starts a separate Java process (jmx-collector.jar) that can’t run without a JRE.
  2. Even with Java, only predefined metrics (JVM, Kafka, Tomcat) would be available. Denodo-specific MBeans aren’t covered.

So: Plan B.

The Solution: JMX Exporter as Java Agent

The Prometheus JMX Exporter solves both problems elegantly:

  • Runs inside the Denodo JVM itself (Java is already there)
  • Has direct access to all MBeans (no remote JMX auth needed)
  • Exposes metrics as an HTTP endpoint on localhost:8081/metrics
  • The CloudWatch Agent scrapes this endpoint via the Prometheus protocol

Architecture

Implementation

Config Files Without Building a New Docker Image

Before we get to the JMX Exporter, a quick detour on a pattern I used here that keeps coming up in other projects since.

The problem: Typically, Docker images (the Java app) and the infrastructure description live in different repos. In my opinion, configurations belong to the IaC side because you want different settings per environment. Baking everything into a Docker image feels wrong. Building a new image for every config change? No thanks.

My solution: A generic preinit script that writes config files from environment variables to disk at container startup. The script goes into the image once, and on the infrastructure side you control which files land where via environment variables.

One caveat: ECS task definitions have a size limit for environment variables. For small configs like the JMX Exporter YAML that’s no issue, but for larger files you’ll hit the ceiling eventually. In that case, S3 + an init script or AWS Systems Manager Parameter Store would be the better choice. For our use case the env var pattern is more than sufficient though.

Denodo Extensions via EFS

Denodo needs various extensions (JDBC drivers, custom wrappers, etc.) that aren’t included in the base image. On ECS Fargate you can’t keep local volumes persistent, so we mount an EFS filesystem into the container. The extensions live there centrally and are available immediately at startup. The nice thing is you can update extensions without deploying a new image or restarting the container.

Activating the JMX Exporter

The JMX Exporter is loaded via JVM opts as a Java agent:

-javaagent:/path/to/jmx_prometheus_javaagent.jar=8081:/tmp/jmx-exporter-config.yaml

The config file is written at runtime by the preinit script (write_config_files.sh) from an environment variable. No extra volume, no init container needed:

write_config_files.sh:

#!/bin/bash
# Generic preinit script: writes config files from environment variables to the filesystem.
#
# Usage: Set WRITE_FILES as a JSON array of objects with "path" and "env" keys.
# Each entry reads the content from the named environment variable and writes it to the specified path.
#
# Example:
#   WRITE_FILES='[{"path":"/tmp/jmx-exporter-config.yaml","env":"JMX_EXPORTER_CONFIG"}]'
#
# This avoids creating a new preinit script for every config file.

if [ -z "${WRITE_FILES:-}" ]; then
  exit 0
fi

if ! command -v jq &>/dev/null; then
  echo "[write-config] ERROR: jq is required but not installed" >&2
  exit 1
fi

errors=0

while IFS= read -r entry; do
  path=$(echo "$entry" | jq -r '.path // empty') || { echo "[write-config] ERROR: failed to parse path from: $entry" >&2; errors=$((errors+1)); continue; }
  env_name=$(echo "$entry" | jq -r '.env // empty') || { echo "[write-config] ERROR: failed to parse env from: $entry" >&2; errors=$((errors+1)); continue; }

  if [ -z "$path" ] || [ -z "$env_name" ]; then
    echo "[write-config] WARN: skipping entry with missing path or env: $entry" >&2
    continue
  fi

  content=$(printenv "$env_name" 2>/dev/null) || true
  if [ -n "$content" ]; then
    mkdir -p "$(dirname "$path")"
    printf '%s\n' "$content" > "$path"
    echo "[write-config] Written $path from \$$env_name"
  else
    echo "[write-config] WARN: \$$env_name is empty or not set, skipping $path" >&2
  fi
done < <(echo "$WRITE_FILES" | jq -c '.[]')

if [ $errors -gt 0 ]; then
  echo "[write-config] ERROR: $errors entries failed to process" >&2
  exit 1
fi

In your CDK/Terraform code it looks like this:

environment: {
  JMX_EXPORTER_CONFIG: jmxExporterConfig,
  WRITE_FILES: JSON.stringify([
    { path: '/tmp/jmx-exporter-config.yaml', env: 'JMX_EXPORTER_CONFIG' }
  ]),
}

CloudWatch Agent Sidecar

The CW Agent runs as a sidecar container in the same task. Since ECS Fargate uses awsvpc networking, all containers share the network interface. localhost:8081 is directly reachable, no service discovery or extra networking required.

CW Agent Config (cloudwatch-agent-config.json):

{
  "logs": {
    "metrics_collected": {
      "prometheus": {
        "cluster_name": "${CLUSTER_NAME}",
        "log_group_name": "/ecs/denodo/prometheus",
        "prometheus_config_path": "env:PROMETHEUS_CONFIG_CONTENT",
        "emf_processor": {
          "metric_namespace": "Denodo/VirtualDataport",
          "metric_declaration": [
            {
              "source_labels": ["job"],
              "label_matcher": "^jmx-exporter$",
              "dimensions": [["service_name"]],
              "metric_selectors": [".*"]
            }
          ]
        }
      }
    }
  }
}

Prometheus Scrape Config (prometheus-config.yaml):

global:
  scrape_interval: 60s
  scrape_timeout: 10s

scrape_configs:
  - job_name: jmx-exporter
    static_configs:
      - targets:
          - localhost:8081
        labels:
          service_name: ${SERVICE_NAME}

The service_name label is set dynamically per environment (virtual-dataport-dev, -tst, -prd) so metrics don’t get mixed up in CloudWatch.

Pitfalls

Of course not everything worked on the first try. Here are the traps I fell into.

CW Agent Version 1.300067.x and the Prometheus Config Format

This agent version has two parsers internally:

  • Target Allocator (OTel-based): Tries to parse the Prometheus YAML as prometheusreceiver.Config and warns: 
    WARN: 'prometheusreceiver.Config' has invalid keys: global, scrape_configs
  • Prometheus Scrape Manager: Uses the standard format and works correctly

The warning is harmless. The Scrape Manager loads the config successfully afterwards. We verified this by running the agent locally in Docker:

docker run --rm \
  -e AWS_REGION=eu-central-1 \
  -e CW_CONFIG_CONTENT='...' \
  -e PROMETHEUS_CONFIG_CONTENT='...' \
  amazon/cloudwatch-agent:1.300067.0b1404

Still, I spent the first half hour thinking this warning was the actual error. It wasn’t.

Missing Required Fields

The agent needs log_group_name and cluster_name in the config. Without log_group_name, the config translation breaks:

prometheus does not have log group name

In ECS, cluster_name is normally auto-detected, but setting it explicitly is safer.

Metrics Not Distinguishable

Without an additional label, all VDP instances have instance=localhost:8081 as their only dimension. Makes sense when you think about it. The fix: set a static service_name label in the Prometheus static_configs and use it as a CloudWatch dimension.

Exit Code 137 on Deployments

During deployments, ECS stops the old task. Denodo takes a while to shut down (license release, closing DB connections). If the stopTimeout (default 30s) is exceeded, SIGKILL hits (exit 137). This isn’t a crash. A stopTimeout: 120s gives Denodo enough time for a clean shutdown. Seeing it for the first time still causes a brief moment of panic though.

Result

After the setup, all JVM and Denodo metrics land in the CloudWatch namespace Denodo/VirtualDataport, cleanly separated by environment:

Metric CategoryExamples
Denodo Serverdenodo_server_activerequests, denodo_server_waitingrequests, denodo_server_activeconnections
JVM Heapjvm_heap_memory_used_bytes, jvm_heap_memory_max_bytes
GCjvm_gc_collection_seconds_sum, jvm_gc_collection_seconds_count
Threadsjvm_threads_current, jvm_threads_deadlocked

The metrics can be visualized directly in CloudWatch dashboards or pulled into Grafana Cloud via a CloudWatch datasource.

AlarmConditionWhy
Heap Memory High> 85% of max over 5 minOOM risk
GC ThrashingGC time > 10% of wall-clock timePerformance degradation
Thread Deadlockjvm_threads_deadlocked > 0Immediately critical
Waiting Requests> 10 over 3 minOverload
JMX Scrape Error= 1 over 2 minMonitoring itself is broken

Conclusion

The detour via JMX Exporter + Prometheus scrape is the only reliable way to get JVM monitoring on ECS Fargate. The native JMX integration of the CW Agent only works on EC2 or in containers with their own JRE. The extra effort (sidecar container, config management) pays off through full visibility into the JVM and Denodo internals.

The preinit script pattern has proven itself in several other services since then, by the way. Write it once, reuse it everywhere.

Back to all posts