OpenNMS Prometheus Remote Writer · v0.4.0

prometheus-remote-writer implements the OpenNMS TimeSeriesStorage SPI from opennms-integration-api v2.0. When OpenNMS Horizon is configured with org.opennms.timeseries.strategy = integration and this plugin is the only TSS implementation registered, every collected sample flows through the plugin to a Prometheus-compatible Remote Write endpoint of your choice.

The plugin pushes OpenNMS resource context — node identity, foreign-source qualification, surveillance categories, interface descriptors, optional metadata — to the backend as native Prometheus labels. Operators query the resulting time series with PromQL directly, from Grafana’s native Prometheus data source. No OpenNMS-side query plugin and no round-trip to the OpenNMS REST API are required at query time.

The KAR will install cleanly into /opt/sentinel/deploy/ and the plugin’s TimeSeriesStorage OSGi service will register, but no samples will reach it: OpenNMS upstream has no Core → Sentinel sample-dispatch path for OIA TSS plugins, and the Sentinel-side streaming-telemetry adapter pipeline (architecturally compatible in principle) has not been verified end-to-end with this plugin.

If you need to offload sample persistence to a Sentinel container today, this is not the right plugin yet. Install it on Horizon Core and use the rest of this guide.

OpenNMS ships a Prometheus integration today via opennms-cortex-tss-plugin. That plugin writes numeric samples to a Prometheus backend but keeps OpenNMS resource context (node label, foreign source, categories, asset record, interface descriptors) in a separate OpenNMS key-value store. To turn opaque resourceId labels back into human-readable resources, you need the OpenNMS Plugin for Grafana, which round-trips to the OpenNMS REST API at query time.

This plugin fixes that at the write path. Resource context is pushed to the backend as first-class Prometheus labels, so PromQL — on any vanilla Grafana Prometheus data source — works end-to-end with no OpenNMS query-time dependency.

The plugin is a clean-room implementation, written from public specifications:

The normative requirement set lives in the project’s tss-plugin spec under openspec/specs/tss-plugin/spec.md.

The plugin’s OSGi bundle is compiled to Java 17 bytecode to match Horizon’s runtime; running on an older JVM fails feature resolution at install time.

The plugin ships as a Karaf KAR (prometheus-remote-writer-kar-X.Y.Z.kar). Drop it into Karaf’s deploy directory and Karaf hot-installs it:

Confirm from the Karaf shell:

Set the time-series strategy in etc/opennms.properties.d/timeseries.properties:

Restart Core for the strategy switch to take effect. The next collector flush sends samples through the plugin.

Drop the bare minimum at etc/org.opennms.plugins.tss.prometheus-remote-writer.cfg:

read.url is the backend’s Prometheus-compatible root. The plugin appends /api/v1/series and /api/v1/query_range itself — do not include /api/v1 in the configured value. See Configuration reference for the full list of knobs and backend-specific URL shapes.

From the Karaf shell:

opennms:prometheus-writer-stats prints all plugin counters and gauges. Watch samples_written_total tick up as OpenNMS pushes its first samples through the plugin.

The repository’s e2e/ directory contains a Docker Compose stack that stands up an OpenNMS Horizon core, Grafana, and one Prometheus-compatible backend of your choice. It is the quickest way to see the plugin work end-to-end on a laptop.

Prerequisites: Docker 24+ with Compose v2.

The fastest way to prove the plugin works is make smoke:

For interactive exploration, see e2e/README.md in the source tree.

Used when wal.enabled=false (the default). When the WAL is enabled, queue.capacity is ignored.

OpenNMS metadata is opt-in only. The built-in denylist always applies.

Opt-in via wal.enabled=true. See Write-Ahead Log for the full operator model.

The full set of normative scenarios lives in openspec/specs/tss-plugin/spec.md in the source tree — that file is the authoritative source of truth for parser behavior, defaults, and edge cases.

The v2 wire layer in this release leaves these fields empty. A future change can populate any of them without breaking the wire layer.

Segment files are named by start offset (00000000000000000000.seg, 00000000000067108864.seg, …​). Each segment has a companion .idx jsonl summary. checkpoint.json at the WAL root tracks the last offset confirmed shipped; written atomically (tmp + fsync + rename) on every advance.

When wal.enabled=true, the plugin scans wal.path at startup and replays any samples whose offset is greater than or equal to checkpoint.json.last_sent_offset. Recovery tolerates torn tails — an incomplete frame at the end of the most recent segment, typical of a process killed mid-append — by logging a WARN, truncating the segment to the last good frame, and resuming.

If any recovery step fails (unwritable path, corrupt checkpoint.json, unreadable segment), the plugin refuses to start with an actionable error message naming the file and the failure reason. Operators reset by removing the WAL directory while the plugin is stopped:

The plugin recreates the directory on next start.

The WAL stores MappedSample (pre-wire). Flipping wire.protocol-version while the WAL holds pending samples is safe — the next flush emits in the new version with no WAL migration step. See Wire protocols (v1 and v2).

The default stays wal.enabled=false. A later release may flip the default once the feature has soaked in real deployments.

The WAL adds these counters and gauges (visible via opennms:prometheus-writer-stats):

The general write counters (samples_written_total, samples_dropped_4xx_total, samples_dropped_5xx_total, samples_dropped_transport_total) are unchanged and continue to apply.

For every sample, the plugin emits the following Prometheus labels when the corresponding source data is available:

Deliberately excluded by default — available via labels.include if you want them: if_alias (user-editable, churns), sys_descr, sys_object_id, asset-record fields, OpenNMS metadata (see Configuration reference for the metadata gating rules).

The default label allowlist is the write-side policy: it decides which OpenNMS-attached tags become Prometheus labels. The read-side — OpenNMS attaching those tags to samples in the first place — lives in OpenNMS itself, and it is off by default.

OpenNMS’s MetaTagDataLoader runs each configured value template through the MATE interpolator against the sample’s scope and attaches a tag for every non-empty result. The property names below and the MATE scope syntax (${node:…}, ${interface:…}, ${service:…}, ${asset:…}) are as of OpenNMS Horizon 35; upstream changes may rename properties in future Horizon releases. If you don’t configure any org.opennms.timeseries.tin.metatags.tag. properties, samples arrive at this plugin carrying only name and resourceId* — and your Prometheus series will have bare {resourceId="…"} labels regardless of what this plugin is configured to emit.

Running more than one OpenNMS instance against the same Prometheus-compatible backend? Two independent knobs exist, and they solve different problems:

Both labels.rename and labels.copy produce a label under a new name, but the mental model differs:

They run in a fixed pipeline:

Copy is one-pass (sees labels that exist at its stage entry; does not recurse) and operates on pre-rename names. Reserved-target rules apply symmetrically to both — a to value that collides with a default label name, a reserved prefix (onms_cat_*, onms_meta_*), another rename target, or another copy target is rejected at startup with an actionable error.

Since 0.2.0 the plugin emits job and instance as default labels so dashboards that compose OpenNMS data with node-exporter, OTel, or other Prometheus data sources in the same backend can use the standard idiom:

The plugin’s instance value carries the OpenNMS-managed device identity (<foreignSource>:<foreignId> when requisitioned, or the numeric dbId), whereas node-exporter emits instance="<host>:<port>" — same label name, different value shapes for the same physical device. Cross-source value correlation (same label value across sources for the same device) requires backend relabel_config; the shared label name alone doesn’t bridge value shapes. job is the primary cross-source scoping filter.

The job value is derived from each sample’s resourceId pattern:

Set job.name = <constant> in the cfg to override the derivation with a fleet-wide constant value (useful when you want every sample from one plugin instance under the same job, e.g., job.name = opennms-prod).

The plugin rejects labels.rename and labels.copy entries whose target would silently clobber an already-emitted label. Reserved targets:

Duplicate rename targets (foo → cluster, bar → cluster) and duplicate from keys (a → cluster, a → tenant) are also rejected. When multiple rename or copy entries have errors, the plugin reports all of them in one startup error so you fix once and restart once.

The plugin sanitizes every metric name, label name, and label value to conform to the Prometheus text model before serialization:

The plugin appends /api/v1/series and /api/v1/query_range itself — do not include /api/v1 in the configured value.

tenant.org-id sets the X-Scope-OrgID header. Behavior per backend:

See Labels and enrichment for the instance.id vs tenant.org-id decision matrix when running multiple OpenNMS instances against the same backend.

The plugin exposes internal operational metrics via a Dropwizard registry and prints them through the Karaf shell command opennms:prometheus-writer-stats.

These are starting points — tune to your deployment’s noise floor.

Set in etc/org.ops4j.pax.logging.cfg or via the Karaf shell (log:set DEBUG org.opennms.plugins.tss.prometheusremotewriter).

These are rough rules-of-thumb, not hard requirements. Measure your deployment.

WAL footprint scales with outage tolerance: at 10k samples/sec each sample averages roughly 30–80 bytes pre-snappy (the wire format is compressed; the WAL is not). Plan for ~30 MB/s of disk in-flight if you want to survive a 10-minute outage.

The plugin’s metrics are exposed via Dropwizard, which means you can scrape the Karaf JVM with a JMX exporter and feed those self-metrics back into the same Prometheus backend the plugin writes to. That gives you one Grafana dashboard with both OpenNMS data and the plugin’s own operational health.

A future release may add a built-in HTTP scrape endpoint to remove the JMX-exporter step. Not on the v0.4 roadmap.

Karaf shell shows the bundle as Failure or unresolved.

The endpoint is rejecting the format. This is usually one of:

samples_dropped_4xx_total increments by the rejected batch size each time. If you’re seeing a steady rate, the deployment isn’t recovering on its own — fix the underlying cause.

The plugin retries 5xx responses with exponential backoff up to retry.max-attempts (default 5), then drops. The same batch is re-enqueued only when wal.enabled=true — without the WAL, exhausted 5xx batches are gone.

If the outage is short enough to fit in wal.max-size-bytes, enabling the WAL turns 5xx outages into delivery delays instead of drops. See Write-Ahead Log.

Three usual culprits:

labels.include = * is rarely correct in production — it surfaces every non-default source tag. The default allowlist is deliberately narrow.

Samples whose resourceId matches none of the parser grammars (bracketed, slash-FS, slash-DB) fall through to job="opennms". The samples_unparseable_resource_id_total counter tracks the rate.

A rising counter is a signal that:

Either way, file an issue with example resourceId strings — the parser is the project’s responsibility, not the operator’s.

Symptom: wal.enabled=true, but every restart starts from an empty WAL and you’ve lost the durability guarantee.

Cause: containerised Karaf with no mounted volume. ${karaf.data} is ephemeral — the default wal.path evaporates on container restart.

Fix: set wal.path explicitly to a path that’s mounted from a persistent volume.

The plugin emits a WARN on startup and every hour when tls.insecure-skip-verify=true. Search for that log line:

Production deployments should always have a valid CA chain. Use tls.ca-file to point at a private bundle if your backend uses an internal CA.

This is an incubation project — community channels first; there is no commercial support yet.

Include with any report:

One compose file per backend — nothing else to match up. The base services (postgres, core, grafana) are defined once in compose.base.yml; each backend file extends: them and appends the backend-specific plugin cfg and Grafana datasource mounts.

First boot of the core container can take several minutes while OpenNMS creates the database and loads features. Watch for:

Grafana auto-provisions a datasource pointing at whichever backend is active (selected by the compose file you brought up). Open Explore → OpenNMS (<backend>) to run PromQL against the data OpenNMS just wrote.

The smoke harness lives entirely in the project Makefile. Per project convention, CI invokes make smoke directly.

Each backend is brought up, polled for > 0 ingested series, and torn down. The target depends on kar, so a fresh KAR is built first. Pass/fail summary is printed at the end; on a timeout, the last 40 lines of the relevant container’s karaf.log are dumped before teardown.

Inside Karaf:

For what opennms:prometheus-writer-stats reports, see Operations.

Once OpenNMS has collected a few samples (default interval: 5 minutes on a fresh provisioning), query the backend directly.

Prometheus:

Mimir (requires the X-Scope-OrgID header; tenant is e2e per the cfg):

VictoriaMetrics:

Use the same -f you brought the stack up with:

-v removes the named data volumes (postgres, opennms, prometheus, mimir, vm). Drop -v if you want to keep state across restarts.

The assembly/kar/target directory is mounted read-only into /opt/opennms/deploy/. A rebuild of the KAR (make kar from the repo root) does not auto-reload the plugin — Karaf’s hot-deploy watches file timestamps, but the container sees the mount at a point in time. To reload a freshly built KAR:

Or, inside the Karaf shell, feature:uninstall + feature:install cycles the plugin without restarting the container.