Deploy Secure Mesh Site v2 with OpenShift Virtualization (ClickOps)

Objective

This guide provides instructions on how to create a Customer Edge (CE) Site using the F5® Distributed Cloud Console for the OpenShift Virtualization environment.

Important: This guide does not provide instructions on how to deploy an F5® App Stack Site.

Planning

Read the following documents before deploying a Secure Mesh Site in any provider environment:

• Understanding F5 Distributed Cloud - Customer Edge (CE)
• CE Datasheet
• CE Supported Platforms Guide
• Customer Edge Site Sizing Reference
• CE Performance Guide: Contact your account representative on CE performance-related information.
• Proxy for CE Registration and Upgrades Reference
• Secure Mesh Sites v2 Frequently Asked Questions
• Customer Edge Registration and Upgrade Reference
• F5 Customer Edge IP Address and Domain Reference for Firewall or Proxy Settings

General Prerequisites

The following general prerequisites apply:

• A Distributed Cloud Services Account. If you do not have an account, see Getting Started with Console.

• Resources required per node: Minimum 8 vCPUs, 32 GB RAM, and 80 GB disk storage. For a full listing of the resources required, see the Customer Edge Site Sizing Reference guide. All the nodes in a given CE Site should have the same resources regarding the compute, memory, and disk storage. When deploying in cloud environments, these nodes should use the same instance flavor.

• Customer Edge (CE) deployments require connectivity to F5 Distributed Cloud. See the F5 Customer Edge IP Address and Domain Reference for Firewall or Proxy Settings guide for the list of IP addresses and domain names that need to be allowed.

• F5 assumes that an existing IPv4 subnet exists with Internet connectivity to attach to the node.

• The new Secure Mesh Site workflow enables you to have up to eight interfaces. However, these interfaces should be in different subnets. Therefore, make sure you have the required subnets available before creating the CE Site nodes.

• If you are deploying the CE site with High Availability (HA) enabled, Internet Control Message Protocol (ICMP) must be opened between the CE nodes on the Site Local Outside (SLO) interfaces. This is needed to ensure intra-cluster communication checks.

Important: After you deploy the CE Site, the IP address for the SLO interface cannot be changed. Also, the MAC address cannot be changed.

OpenShift Prerequisites

• Ensure you have a functioning OpenShift cluster with OpenShift Virtualization installed, with v4.20.14.

• Ensure you have the necessary permissions to configure compute and network settings on your OpenShift Virtualization cluster.

Configuration Overview

To create a Secure Mesh Site with OpenShift, here are the high-level steps:

• Site object configuration: Create and configure a Secure Mesh Site object using F5 Distributed Cloud Console.
• Node creation prerequisites: Create objects that are associated with the CE nodes, including a network security group, networks, volume, and more.
• Image management: Use the Copy Image Name option to generate a URL to download the image file. Alternately, use the Download Image option to download the image file locally to your system.
• Node management: Use the image to launch CE nodes. Each node is a virtual machine (VM).

Important: The first interface of a CE node must be mapped to the Site-Local Outside (SLO) VRF which should allow connectivity to the F5 Distributed Cloud.

The document describes one- and two-interface deployments for CE sites with HA disabled (single node) and HA enabled (three-node cluster).

Configure OpenShift Virtualization

To optimize virtual machine performance, enable the CPU Manager and configure the Topology Manager on your OpenShift cluster. This ensures resource allocation efficiency and keeps VM resources within the same non-uniform memory access (NUMA) domain.

To enable, see the official documentation.

Create Site Object

• Create a secure mesh site object in Distributed Cloud Console. Refer to the Create Secure Mesh Site guide.

• Set the Provider Name option to OpenShift Virtualization.

• For High Availability, choose an option. If it is Disabled, then the CE Site only supports one node. If it is Enabled, the CE Site requires three nodes. Additional nodes can only be added to CE sites when HA is Enabled.

Important: The High Availability mode cannot be changed after the CE Site is created.

• Leave the other options with default values. These options have intelligent default values and do not need further configuration. Refer to the Create Secure Mesh Site guide for more information on these options.

• Click Add Secure Mesh Site.

Generate Node Token

A one-time node token is required to register a CE Site node to the Distributed Cloud Console. A new token must be generated for every new node in a CE Site. A token is valid for 24 hours. Make sure that the CE node is deployed soon after the token is generated.

The token is included in the cloud-init information under the Content variable. Also included are two variables commented out: slo_ip and slo_gateway. These variables can be commented out if you are using your own DNS service and not the default DNS service provided by F5.

• In Distributed Cloud Console, select the Multi-Cloud Network Connect workspace.

• Navigate to Manage > Site Management > Secure Mesh Sites v2.

• For your site, click ... > Generate Node Token.

• Click Copy cloud-init.

• Save the value locally. This token is used later.

• Click Close.

• Generate one token per node you intend to deploy.

Create User Data File

Create a user-data.txt file with the node token.

• In the token variable, paste the node token information generated from the section above.

#cloud-config
write_files:
- path: /etc/vpm/user_data
content: |
token: <your-node-token>
owner: root
permissions: '0644'

• Optionally, configure a static IP address for the SLO interface. To assign a static IP address to the slo interface, add the following lines to the user-data.txt file:

slo_ip: <IP>/<prefix>
slo_gateway: <GW IP>
slo_dns: <DNS IP>

Create DataVolume Manifest for Importing CE Node Image

• Create a new DataVolume manifest to import the CE node image into your OpenShift cluster.

• For your Site object, under Actions, click ... > Copy Image Name to receive a download link to use.

• Paste the URL link in the url variable.

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: cev2-2-datavolume
namespace: f5-ce
spec:
source:
http:
url: "http://<http server address>/f5xc-ce-9.2024.22-
20240806132626.qcow2"
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 150Gi

Create VM manifest File

Use the CLI to create and deploy the VM manifest. Once the manifest file is applied, the CE node is deployed.

Before deploying a new virtual machine in OpenShift, find your right size and requirements by referring to the Customer Edge Site Sizing Reference guide.

Important: The name of the VM should not have "." in it. For example, the hostname can be node-0 or node0, but it cannot be node.f5.com since it is not supported. Your node VM name must adhere to DNS-1035 label requirements. This means the name must consist of lower case alphanumeric characters or “-“, start with an alphabetic character, and end with an alphanumeric character.

If configuring a multi-node site, each node hostname must be unique.

• Create a new OpenShift project to house the namespaced resources.

oc new-project f5-ce

• Create the Network Attachment Definitions (NADs).

• Apply the NADs.

oc apply -f ce-internal.yaml -f ce-external.yaml

• Create the VM manifest.

• Apply the VM manifests to create the CE nodes (instances).

oc apply -f cev2-2-datavolume.yaml -f cev2-2-vm.yaml

• In a separate terminal, monitor the VM creation process:

oc get virtualmachines -n f5-ce

oc get pods -n f5-ce

Example VM Manifest

Full Example VirtualMachine resource manifest (cev2-2-vm.yaml)
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: cev2-2
namespace: f5-ce
labels:
app: f5-ce
spec:
runStrategy: Always
template:
metadata:
labels:
app: f5-ce
spec:
F5 XC CE in OpenShift virtualization deployment guide 10
domain:
resources:
requests:
memory: 32Gi
cpu: "8"
devices:
disks:
- name: f5-ce-disk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- name: outside-net
bridge: {}
- name: inside-net
bridge: {}
networks:
- name: outside-net
multus:
networkName: "f5-ce/ce2-slo"
- name: inside-net
multus:
networkName: "f5-ce/ce2-sli"
volumes:
- name: f5-ce-disk
dataVolume:
name: cev2-2-datavolume
- name: cloudinitdisk
cloudInitNoCloud:
networkData: |
network:
version: 2
ethernets:
enp1s0:
dhcp4: false
addresses: [192.170.10.162/24]
gateway4: 192.170.10.1
nameservers:
addresses:
- 10.105.134.20
- 10.105.134.21
search:
- bd.f5.com
optional: true
userData: |-
#cloud-config
hostname: master-0
fqdn: master-0
manage_etc_hosts: true
chpasswd:
F5 XC CE in OpenShift virtualization deployment guide 11
list: |
admin:Volterra12345
expire: False
runcmd:
- [ hostnamectl, hostname, node-0]
write_files:
- path: /etc/vpm/user_data
permissions: '0644'
owner: root
content: |
token: <token>Fu
slo_ip: 192.170.10.162/24
slo_gateway: 192.170.10.

Verify CE Site Registration

After you deploy your nodes, they automatically register as a CE Site in Distributed Cloud Console. The registration process is not instantaneous. In Console, the status changes from Waiting for Registration to Provisioning to Online. Wait a few minutes for the registration process to begin after completing the preceding sections.

• In Distributed Cloud Console, navigate to Multi-Cloud Network Connect > Overview > Infrastructure > Sites.

• Select your CE Site. The Dashboard tab should clearly show that the CE Site has registered successfully with the System Health of 100% as well as Data Plane/Control Plane both Up.

Note: For more information on the CE Site registration process, see the Customer Edge Registration and Upgrade Reference guide.

Post-Deployment Configuration - Optional

After your CE Site registers successfully, you might want to add additional network interfaces to meet your requirements. Ensure that you connect another network interface to the node VM.

Important: Adding or removing network interfaces causes the data plane services on the CE node to restart. Therefore, F5 strongly recommends that you perform this operation during maintenance windows. As data plane services restart, traffic drops are expected, as well as tunnels to F5 Distributed Cloud REs going down.

All CE nodes in a given multi-node Site should have the same number of network interfaces attached. CE nodes with non-homogenous interfaces within a CE Site might cause issues.

Each node in a multi-node CE Site should have interfaces with the same VRFs assigned. For example: If a CE Site has three nodes, with each node having two interfaces - the first interface on each node is auto-configured to be in the SLO VRF (to connect to F5 Distributed Cloud). If the second interface on node-1 is in the SLI VRF, then the second interface on node-2 and node-3 must also be in the SLI VRF.

When new interfaces are added, they are auto-discovered. You can configure the interface (for example: place the interface in the appropriate VRF) from the CE Site configuration.

The first interface of the CE nodes should not be removed or modified.

After you configure the SLO interface with a static IP address, DHCP will still be displayed in the Console. However, your static IP configuration is well taken into account. Also, remember that you cannot modify SLO parameters once the node is registered and deployed.

Modify Interface Attributes

Important: The IP address for the SLO interface cannot be modified. In addition, you cannot modify any MAC addresses for any interfaces.

• Power down the VM prior to adding any new interfaces or modifying any existing interfaces.

• To modify any interface attributes, click Manage Configuration.

• Click Edit Configuration.

• Under the Nodes subsection, click the pencil icon under Actions to edit.

• Choose one of the interfaces to edit. This example uses ens6.

• Change the settings as required. In this example, the interface is being placed in the prod-segment. Therefore, the setting from the original Site Local Inside (Local VRF) to Segment (Global VRF) was changed. Then, the required segment is selected.

• Click Apply to apply the interface attribute changes.

• Click Apply again to confirm interface attribute changes.

• Click Save Secure Mesh Site.

• Power back up the node VM.

Day 2 Operations

• To monitor your Site, see the Monitor Site guide.
• To manage your Site software and OS updates, see the Manage Site guide.
• For troubleshooting issues, see the Troubleshooting Guide for Secure Mesh Site v2 Deployment guide. It provides step-by-step instructions to debug and resolve the issues that may arise due to registration and provisioning errors.
• For the latest on Distributed Cloud Services releases, see Changelogs.
• To view the various types of events generated, see the Events Reference guide.

Related Guides

To create a load balancer on the CE Site, see the HTTP Load Balancer or the TCP Load Balancer guides.

Concepts

• System Overview
• Core Concepts
• Networking
• F5 Distributed Cloud - Customer Edge