For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Control plane metrics

Monitor control plane health and performance with Prometheus metrics.

By default, the agentgateway control plane exposes metrics in Prometheus format. You can use these metrics to monitor the health and performance of your gateway environment, or to verify that the control plane is emitting expected metrics when debugging your observability stack. For more information about how metrics are implemented, refer to the kgateway project developer docs.

Before you begin

Enable monitoring with Helm

The agentgateway Helm chart can create the Prometheus and Grafana resources that collect and visualize these metrics. The chart does not create these resources by default.

Before you begin: Install the Prometheus Operator custom resource definitions (CRDs) in your cluster before you enable monitoring. Without the CRDs, Prometheus cannot recognize the ServiceMonitor and PodMonitor resources that the chart creates. The agentgateway installation does not include these CRDs. For one way to install them, along with Prometheus and Grafan as part of an OTel stack, see Set up Prometheus.

Steps to enable monitoring in Helm:

1. Create a monitoring-values.yaml file that enables the monitoring resources.

   monitoring:
     enabled: true

   Review the following table to understand the resources that the chart creates when you set monitoring.enabled to true. To disable or tune an individual resource, set the fields in its own section of the monitoring section.

   Resource	What it scrapes	Section to tune
   ServiceMonitor	The control plane deployment on the metrics port, 9092 by default	monitoring.serviceMonitor
   PodMonitor	The proxy pods for the GatewayClasses in monitoring.proxy.gatewayClassNames on the metrics port, 15020 by default	monitoring.proxy
   ConfigMap for the Grafana dashboard	Nothing. The Grafana sidecar discovers the ConfigMap through the grafana_dashboard: "1" label.	monitoring.grafanaDashboard

   Note

   The PodMonitor selects proxy pods in the agentgateway-system namespace only. Because gateway proxies usually run in the namespace of the Gateway that provisions them, set monitoring.proxy.namespaceSelector to {any: true} to scrape proxies in every namespace, or to a matchNames list to scrape specific namespaces.

2. Upgrade your agentgateway installation with the values file.

   helm upgrade -i -n agentgateway-system agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
   --version 0.0.0-latest-dev \
   -f monitoring-values.yaml

3. Verify that the monitoring resources are created.

   kubectl get servicemonitor,podmonitor,configmap -n agentgateway-system -l app.kubernetes.io/name=agentgateway

   Example output:

   NAME                                                AGE
   servicemonitor.monitoring.coreos.com/agentgateway   3s
   
   NAME                                                  AGE
   podmonitor.monitoring.coreos.com/agentgateway-proxy   3s
   
   NAME                               DATA   AGE
   configmap/agentgateway-dashboard   1      3s
   

Your Prometheus instance now scrapes the control plane and proxy metrics, and Grafana loads the dashboard from the ConfigMap.

View control plane metrics

The following steps show you how to view the raw metrics endpoint of the control plane deployment.

1. Port-forward the control plane deployment on port 9092.

   kubectl -n agentgateway-system port-forward deployment/agentgateway 9092

2. Open your browser to the metrics endpoint: http://localhost:9092/metrics.

   Example output:

   # HELP agentgateway_controller_reconciliations_total Total number of controller reconciliations
   # TYPE agentgateway_controller_reconciliations_total counter
   agentgateway_controller_reconciliations_total{controller="gateway",result="success"} 1
   agentgateway_controller_reconciliations_total{controller="gatewayclass",result="success"} 2
   agentgateway_controller_reconciliations_total{controller="gatewayclass-provisioner",result="success"} 2
   

Control plane metrics reference

Review the following table to understand more about each metric.

Helpful terms:

• Controller: A Kubernetes controller that reconciles resources as part of the agentgateway control plane deployment.

• Resource: A Kubernetes object that is managed by a controller of the control plane.

• Snapshot: A complete, point-in-time representation of the current state of resources that the controller builds and serves to a gateway proxy via the Envoy extensible Discovery Service (XDS) API.

• Sync: The metrics refer to two kinds of syncs:

  • Status sync metrics represent the time it takes for you as a user to view the status that is reported on the resource.
  • Snapshot sync metrics roughly represent the time it takes for a resource change to become effective in the gateway proxies.

• Transform: The process of the control plane converting high-level resources or intermediate representations (IR) into lower-level representations into the structure that the XDS API expects for a snapshot.