Metrics from HashiCorp-managed clusters

This page describes how to scrape Consul server metrics from HashiCorp-managed clusters.

HashiCorp ensures the availability of HashiCorp-managed clusters and provides observability features through HCP Consul Central. You can set up a custom metrics collector to track cluster usage.

For opinionated, preconfigured graphs of Consul server and Envoy proxy metrics, refer to the HCP Consul observability overview.

Prerequisites

To access server metrics, you need a telemetry agent capable of both resolving a DNS record to IP addresses and scraping Prometheus-format metrics. This page provides example configurations for Prometheus and OpenTelemetry Collector using Docker.

Accessing server metrics requires the cluster's address and an ACL token with a minimum of agent:read permission. To get the cluster's address and create an ACL token using HCP Consul, refer to Access HashiCorp-managed clusters.

Scrape server metrics

Configure your telemetry collector to scrape metrics from the cluster's /v1/agent/metrics endpoint and then start the collector. The following examples demonstrate configurations for Prometheus and OpenTelemetry Collector.

Prometheus

To use a Prometheus metrics collector, configure it to do the following:

1. Resolve the cluster's DNS address to IP addresses.
2. Scrape metrics from each server's /v1/agent/metrics endpoint.
3. Call the agent endpoint with a Bearer token authorized to read metrics.

In the following example configuration, these fields are specified so that the collector scrapes agent metrics every 60 minutes by running a job named hcp-consul-cluster.

---
global:
  scrape_interval: "60s"

scrape_configs:
  - job_name: "hcp-consul-cluster"

    # Resolve the IP addresses of servers.
    scheme: "https"
    dns_sd_configs:
      - names:
          - "<consul-cluster-name.consul.alphanumeric-id.aws.hashicorp.cloud>"
        type: "A"
        port: 443

    # Set the Consul token as the Bearer token.
    authorization:
      credentials: "<token>"

    # Query the agent metrics endpoint.
    metrics_path: "/v1/agent/metrics"

    # Disable TLS verification.
    tls_config:
      insecure_skip_verify: true

To start the metrics collector, run the following command:

$ prometheus --config.file=scrape.yaml

OpenTelemetry Collector

You can use the Prometheus Receiver in the OpenTelemetry Collector to scrape metrics from a HashiCorp-managed cluster.

The following example configures a metrics pipeline with a Processor and Exporter. The OTLP Exporter requires an OTLP gRPC endpoint. Observability platforms such as New Relic and Honeycomb have public endpoints for metrics ingestion. Others platforms such as Datadog have their own Exporter.

---
receivers:
  prometheus:
    config:
      global:
        scrape_interval: "60s"
      scrape_configs:
        - job_name: "hcp-consul-cluster"
          scheme: "https"
          dns_sd_configs:
            - names:
                - "<consul-cluster-name.consul.alphanumeric-id.aws.hashicorp.cloud>"
              type: "A"
              port: 443
          authorization:
            credentials: "<token>"
          metrics_path: "/v1/agent/metrics"
          tls_config:
            insecure_skip_verify: true

processors:
  batch:
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: "60s"

exporters:
  otlp:
    endpoint: "<otlp-compatible-endpoint>"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp]

To start the Collector in a Docker container, run the following command:

$ docker run -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.86.0