Link self-managed clusters with HCP Consul overview

This topic describes how to link self-managed Consul clusters to HCP Consul. When you link self-managed clusters to HCP Consul, you grant either read/write or read-only access permissions to HCP Consul Central, enabling centralized visibility and management into your clusters through the HCP Portal.

Introduction

When you use HCP Consul, you can allow HashiCorp to install, configure, and manage Consul servers for you. This arrangement allows you to use HashiCorp-managed Consul clusters with services deployed to your cloud service provider. Using HashiCorp-managed clusters simplifies the process of setting up and using Consul and provides high-level insights into your network through HCP Consul Central.

You can also link your own self-managed Consul clusters, or clusters you install, configure, and manage yourself, to HCP Consul. Using self-managed clusters with HCP Consul can help you streamline network operations while maintaining control over your clusters. It also enables HCP Consul Central features, such as observability dashboards, for your self-managed clusters.

You can use HCP Consul to help you create new self-managed clusters on VMs or Kubernetes that link automatically, or you can link an existing cluster that is already deployed to a cloud or on-prem environment.

How the linking process works

When you initiate the linking process in the HCP portal, HCP Consul generates credentials for you. When you add these credentials to your Consul cluster's agent configuration (VMs) or Helm chart (Kubernetes) and apply the updates to your active cluster, your self-managed cluster automatically authenticates with your HCP organization and connects to HCP Consul Central. The following list identifies the name and purpose of each credential:

• consul-hcp-client-id: Identifies the self-managed cluster to HCP
• consul-hcp-client-secret: Authorizes connection to HCP during the linking process
• consul-hcp-resource-id: Identifies the HCP organization and project associated with the linked cluster

By adding these credentials to your cluster secrets and referencing them in the Cloud stanza of the agent configuration (VMs) or global.cloud stanza of your Helm chart (Kubernetes), your cluster automatically links to HCP Consul. As part of this process, HCP Consul Central creates an ACL token in your cluster that grants it read/write access permissions. You also have the option to use a self-managed read-only token to link your cluster, which enables some centralized management operations while denying HCP Consul write permissions for the cluster. Your selection determines which cluster management features are available to your organization's users through the HCP Portal. Refer to Cluster access permissions for more information.

The following diagram describes the Cloud configuration stanza's role in linking self-managed environments with HashiCorp-managed environments. The HashiCorp-managed Vault securely stores the necessary secrets, including the token that provides HCP Consul Central access to the cluster.

For more information about configuring the cloud stanza, including technical specifications for its parameters, refer to the self-managed HCP parameters in the Consul documentation's agent configuration reference.

Stored secrets

The management token generated during the linking process is stored in a HashiCorp-managed Vault with other secrets that become managed once the link is established. The full list of managed secrets include:

• A management token, which is an ACL token with either a read/write or a read-only policy attached to it.
• The gossip key used for encrypted communication between agents.
• The TLS certificate authority (CA) that Consul uses to encrypt RPC communication between service mesh proxies.
• The TLS certificates issued by the TLS CA. The TTL on the certificates is one year.

Access a linked cluster

When clusters are not linked to HCP Consul, accessing the UI for multiple Consul servers requires managing multiple ACL tokens that are separate from your HCP credentials. However, HCP Consul Central enables access to a linked cluster’s UI directly through your web browser without additional credentials. It provides an endpoint for you to access a server, then retrieves the required ACL token from the managed Vault environment so that your Consul server authorizes access to its UI.

When ACLs are enabled in the linked cluster, the permissions for particular users or groups of users to access the Consul UI through HCP must be separately managed by an administrator in your HCP organization. Refer to user permissions for more information about managing users and their permissions in HCP.

HCP user roles and ACL policies

When linking a self-managed cluster to HCP Consul in read/write mode, HCP Consul Central accesses a token with the global-management ACL policy. It also creates two additional ACL policies in the self-managed cluster that support access according to a user's role in the HCP organization. The following table summarizes the access permissions for each user role:

The consul-management-plane-ui-read-write policy contains the following ACL rules for accessing the cluster:

mesh = "read"
peering = "read"
operator = "read"

agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "write"
  intentions = "read"
}

kv_prefix "" {
  policy = "write"
}

key_prefix "vault/" {
  policy = "deny"
}

service "vault" {
  policy = "read"
}

The consul-management-plane-ui-read-only policy contains the following ACL rules for accessing the cluster:

mesh = "read"
peering = "read"
operator = "read"

agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "read"
}

After your self-managed cluster is successfully linked, HCP Consul Central may take up to 10 minutes to create these policies and role assignments. Only HCP users in the organization's admin role can access the cluster using the HCP Portal during this time. Other users in your organization will receive access after the ACL policies attached to their tokens are created on the self-managed cluster.

Benefits

• Unified service catalog: HCP Consul Central collects service instance location and health information in a single aggregated service catalog. Search for a service by name and then find clusters where service instances are deployed.
• Cluster overview: Visualize and monitor clusters, including service counts, versions, and health. Get more information about any cluster, deploy clusters, and establish cluster peering connections using a single interface.
• Server and proxy observability: Get additional observability into clusters and their performance metrics with visual representations of telemetry information. Available metrics include server heartbeat and proxy health. Refer to Consul observability for more information.
• Global workflows: Streamline the process to establish cluster peering connections between clusters linked to HCP Consul Central using a dedicated workflow in the HCP UI.

Workflow

You can use HCP Consul to create and link self-managed clusters to HCP Consul Central. Whether you use Consul with VMs or Kubernetes, the workflow to create and link existing clusters follows a similar sequence.

After you begin the process in HCP Consul, HCP provides you with credentials that you can copy and paste into your local environment. These credentials identify the HCP organization and are used to authenticate the cluster during the linking process. HCP also provides you with a configuration stanza referencing these credentials formatted for your chosen runtime. You can copy and paste this stanza into the Consul agent’s configuration file or Helm chart.

When you restart the server where you applied the configuration or apply the Helm updates to your Kubernetes cluster, the changes take effect and your cluster begins the linking process automatically.

When you initiate the cluster creation process after adding these configuration details, HCP Consul generates a management token with permissions that give it administrative control over your self-managed cluster.

After a few minutes, your newly created cluster connects to HCP Consul Central, making it available in the HCP portal.

Constraints and limitations

Be aware of the following constraints and technical limitations on linking self-managed clusters to HCP Consul:

• HCP Consul does not support linking to self-managed clusters that run Consul's v2 catalog API.