Observability & Monitoring

1. Overview

KOLMOS exposes a hand-rolled Prometheus text-exposition endpoint and structured logging, both off by default so a plain local run stays quiet. Turn them on with `--metrics-port` and `--log-format` on `serve`. Host-level CPU/RAM metrics are deliberately not exported — that's left to `node_exporter`/`cAdvisor`/`docker stats` running alongside the container, so KOLMOS's own `/metrics` stays focused on what only the database itself knows.

2. Prometheus Metrics

`--metrics-port <port>` starts a dedicated HTTP listener (bound to 127.0.0.1 by default) serving `GET /metrics` and `GET /healthz`. `/metrics` is deliberately unauthenticated — there are no secrets in the exported series — so treat the port itself as something to keep off the public internet. Exported series: - `kolmos_connections_active`, `kolmos_connections_total` — labeled by `protocol="pg"|"mysql"|"mongo"` - `kolmos_queries_total`, `kolmos_query_errors_total` — per protocol - `kolmos_query_duration_seconds` — histogram, buckets 0.001s–5s - `kolmos_open_stores` — multi-store mode only - `kolmos_vacuum_runs_total`, `kolmos_vacuum_last_duration_seconds`

kolmos --root /data serve --host 0.0.0.0 --metrics-port 9090

curl http://127.0.0.1:9090/metrics
curl http://127.0.0.1:9090/healthz

3. Example Prometheus Scrape Config

scrape_configs:
  - job_name: kolmos
    static_configs:
      - targets: ["kolmos-host:9090"]
    metrics_path: /metrics
    scrape_interval: 15s

4. Structured Logging

`--log-format {text,json}` on `serve` controls output shape (default `text`); pick `json` to emit one JSON object per line for Loki, CloudWatch, or ELK ingestion. Log verbosity is controlled by the `KOLMOS_LOG` environment variable (not `RUST_LOG`), defaulting to `info`. Logs always go to **stderr**, keeping stdout clean for CLI command output you might pipe or script against.

KOLMOS_LOG=debug kolmos --root /data serve --host 0.0.0.0 --log-format json 2>> /var/log/kolmos.jsonl