Uninstalling NVIDIA Enterprise AI Drivers from ESXi

Added , By with No Comments

This blog post guides you through uninstalling NVIDIA Enterprise AI drivers from an ESXi 8.0U2 host.

Putting the ESXi Host into Maintenance Mode

Before modifying software configurations, it’s crucial to put the ESXi host into maintenance mode. This ensures no running virtual machines are affected during the process.

Checking Installed NVIDIA Drivers

Once in maintenance mode, SSH to the host and use the following command to identify currently installed NVIDIA drivers:

[root@esx1-sree-lab:~] esxcli software vib list | grep -i NVD
NVD-AIE_ESXi_8.0.0_Driver      535.154.02-1OEM.800.1.0.20613240      NVD     VMwareAccepted    2024-03-20    host
nvdgpumgmtdaemon               535.154.02-1OEM.700.1.0.15843807      NVD     VMwareAccepted    2024-03-20    host

The output will display details like driver name, version, and installation date. In the example, the following NVIDIA VIBs are found:

  • NVD-AIE_ESXi_8.0.0_Driver
  • nvdgpumgmtdaemon

Removing the Driver VIBs

Now, proceed to remove the listed VIBs using the esxcli software vib remove command. Here’s how to remove each VIB:

  • nvdgpumgmtdaemon:
[root@esx1-sree-lab:~] esxcli software vib remove -n nvdgpumgmtdaemon
Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   VIBs Installed:
   VIBs Removed: NVD_bootbank_nvdgpumgmtdaemon_535.154.02-1OEM.700.1.0.15843807
   VIBs Skipped:
   Reboot Required: true
   DPU Results:

This command removes the nvdgpumgmtdaemon VIB. The output will confirm successful removal and indicate a required reboot for changes to take effect.

  • NVD-AIE_ESXi_8.0.0_Driver:
[root@esx1-sree-lab:~] esxcli software vib remove -n NVD-AIE_ESXi_8.0.0_Driver
Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   VIBs Installed:
   VIBs Removed: NVD_bootbank_NVD-AIE_ESXi_8.0.0_Driver_535.154.02-1OEM.800.1.0.20613240
   VIBs Skipped:
   Reboot Required: true
   DPU Results:

Similarly, this command removes the main NVIDIA driver VIB and prompts for a reboot.

Rebooting the ESXi Host

After removing both VIBs, it’s essential to reboot the ESXi host to apply the changes. Use the following command:

[root@esx1-sree-lab:~] reboot

The host will reboot, and the NVIDIA drivers will be uninstalled.

Verifying Uninstallation

Once the ESXi host restarts, confirm that the NVIDIA drivers are no longer present. Use the same command as before to check the installed VIBs:

[root@esx1-sree-lab:~] esxcli software vib list | grep -i NVD

If the output is empty or doesn’t contain any NVIDIA-related entries, the drivers have been successfully uninstalled.

Important Notes:

  • This guide serves as a general overview. Always refer to the official documentation for your specific NVIDIA driver version and ESXi host configuration for detailed instructions.
  • Putting the ESXi host into maintenance mode is crucial to avoid disruptions to running virtual machines.

By following these steps, you can effectively uninstall NVIDIA Enterprise AI drivers from your ESXi 8.0U2 host

Updating the Harbor Certificate on Your TKG Cluster (vCD 10.5.1, CSE 4.2)

Added , By with No Comments

This blog post guides you through replacing the certificate for your Harbor registry deployed on a Tanzu Kubernetes Grid (TKG) cluster using Helm charts. We’ll assume you’re using VCD version 10.5.1 and Container Service Extension (CSE) version 4.2.

Understanding the Need for Certificate Replacement

Harbor certificates, like any security certificate, may need to be replaced due to expiration, security upgrades, or changes in your PKI infrastructure. This process ensures secure communication within your container registry.

Prerequisites

  • Access to your TKG cluster and kubectl CLI.
  • New certificate and key files (harbor-v2.crt and harbor-v2.key).

Steps:

  1. Create a New Secret:
  • We’ll store the new certificate and key in a Kubernetes secret for secure management. Use the kubectl create secret tls command to create a secret named harbor-secret-v2:
   kubectl create secret tls harbor-secret-v2 --cert=harbor-v2.crt --key=harbor-v2.key --namespace=harbor-system

Replace harbor-v2.crt and harbor-v2.key with the actual filenames of your certificate and key files.

  1. Update the values.yaml file:
  • The values.yaml file defines various configurations for your Harbor deployment. Locate this file from your initial Harbor deployment process.
  • Edit the values.yaml file to point to the newly created secret. Look for the certSource section and update it to use secrets:
   certSource: secret
   secret:
     secretName: "harbor-secret-v2"

This configures Harbor to use the certificate and key stored in the harbor-secret-v2 secret.

  1. Upgrade Harbor with the New Configuration:
  • Deploy the updated configuration with the new certificate using the helm upgrade command:
   helm upgrade harbor harbor -n harbor-system -f ./values.yaml

This command upgrades the harbor deployment in the harbor-system namespace using the configuration specified in the updated values.yaml file.

Conclusion

By following these steps, you’ve successfully replaced the certificate for your Harbor registry deployed on your TKG cluster. Remember to update your Harbor clients or local configurations to reflect the new certificate details for continued secure communication.

How to Install NVIDIA Enterprise AI Driver on VMware ESXi

Added , By with No Comments
Reply

In the realm of modern data centers, leveraging GPU acceleration has become a crucial aspect of enhancing computational capabilities, especially in AI and machine learning workloads. VMware ESXi, a leading hypervisor, coupled with NVIDIA’s AI Enterprise software stack, empowers enterprises to efficiently deploy and manage GPU-accelerated virtualized environments. In this guide, we’ll walk you through the process of installing the NVIDIA Enterprise AI driver on VMware ESXi, enabling support for the NVIDIA H100 SXM 80GB HBM3 GPU.

Prerequisites:

  • PowerEdge XE9680 server
  • VMware ESXi 8.0 Update 2 installed
  • Custom ESXi image profile: DEL-ESXi_802.22380479-A04
  • NVIDIA H100 SXM5 80GB HBM3 GPU

Great News! NVIDIA Enterprise AI 5.0 now supports the NVIDIA H100 SXM 80GB HBM3, previously limited to bare-metal deployments (details in https://docs.nvidia.com/ai-enterprise/latest/release-notes/index.html#whats-new:~:text=Newly%20supported%20graphics,Grace%20Hopper%20Superchip).

Step 1: Download the Driver Bundle

Firstly, obtain the NVIDIA AI Enterprise 5.0 driver bundle from the NVIDIA License portal. Ensure compatibility with your ESXi version and GPU model. In this case, we are using NVIDIA-AI-Enterprise-vSphere-8.0-550.54.16-550.54.15-551.78.zip.

Step 2: Upload the Driver Bundle

Unzip the downloaded driver bundle and upload it to a shared datastore within your vSphere cluster. Utilize the vSphere client’s File browser option for seamless uploading.

Step 3: Prepare the Host

  1. Put the ESXi host into maintenance mode to ensure uninterrupted installation.
  2. SSH into the ESXi host for command-line access.

Step 4: Install the NVIDIA Enterprise AI Driver

Execute the following command to install the NVIDIA Enterprise AI driver on the ESXi host:

esxcli software vib install -d /vmfs/volumes/<datastore>/path/to/driver_bundle.zip

Replace and path/to/driver_bundle.zip with the appropriate datastore name and path. After installation, you should receive a confirmation message indicating successful installation.

[root@esx03:~]  esxcli software vib install -d /vmfs/volumes/nfs-ds-1/nvidia-550/NVD-AIE-800_550.54.16-1OEM.800.1.0.20613240_23471877.zip
Installation Result
   Message: Operation finished successfully.
   VIBs Installed: NVD_bootbank_NVD-AIE_ESXi_8.0.0_Driver_550.54.16-1OEM.800.1.0.20613240
   VIBs Removed:
   VIBs Skipped:
   Reboot Required: false
   DPU Results:

Step 5: Reboot the Server

Reboot the ESXi host to finalize the driver installation process.

Step 6: Verify Installation

Upon reboot, ensure that the NVIDIA vGPU software package is correctly installed and loaded. Check for the NVIDIA kernel driver by running the following command:

[root@esx03:~]vmkload_mod -l |grep nvidia 

nvidia                   58420

You should see the NVIDIA kernel driver listed among the loaded modules.

Additionally, confirm successful communication between the NVIDIA kernel driver and the physical GPUs by running:

nvidia-smi
[root@esx03:~] nvidia-smi
Tue Mar 26 14:47:43 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.16              Driver Version: 550.54.16      CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:19:00.0 Off |                    0 |
| N/A   35C    P0             77W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:3B:00.0 Off |                    0 |
| N/A   31C    P0             77W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          On  |   00000000:4C:00.0 Off |                    0 |
| N/A   30C    P0             76W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          On  |   00000000:5D:00.0 Off |                    0 |
| N/A   34C    P0             75W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          On  |   00000000:9B:00.0 Off |                    0 |
| N/A   34C    P0             78W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          On  |   00000000:BB:00.0 Off |                    0 |
| N/A   31C    P0             77W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          On  |   00000000:CB:00.0 Off |                    0 |
| N/A   33C    P0             78W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          On  |   00000000:DB:00.0 Off |                    0 |
| N/A   31C    P0             78W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
[root@az1-wld-esx01:~]

This command should display information about the installed NVIDIA GPUs, confirming proper functionality.

Step 7: Repeat for Cluster Hosts

Repeat the aforementioned steps for all hosts within your vSphere cluster to ensure consistent GPU support across the environment.

Simplifying VM Deployments with vGPU Policies in VMware Cloud Director 10.5.1

Added , By with No Comments

Virtual Graphics Processing Units (vGPUs) are a game-changer for cloud deployments, enabling high-performance graphics processing for workloads like 3D design, video editing, and AI applications within virtual machines (VMs). VMware Cloud Director (VCD) streamlines vGPU management through vGPU policies, allowing you to define the allocation of these powerful resources to your VMs.

This blog post will guide you through creating a vGPU policy in VCD, ensuring your VMs have the graphics horsepower they need:

Prerequisites:

  • Access to the VCD Provider Portal with administrative privileges.
  • Pre-configured vGPU profiles in VCD. These profiles represent the different types of vGPUs available in your environment, typically created from the capabilities of your underlying vSphere cluster with NVIDIA GPUs.

Creating a vGPU Policy:

  • Log in to the VCD Provider Portal with your administrative credentials.
  • Verify vGPU Profile Visibility: Navigate to Infrastructure Resources > vGPU Profiles. Ensure the vGPU profiles corresponding to your available GPUs are listed here. If not, you’ll need to create them beforehand (refer to your VCD documentation for specific steps).
  • Create the vGPU Policy:
    • Go to Cloud Resources > vGPU Policies.
    • Click New.
    • On the “What is a vGPU Policy?” screen, click Next.
  • Define Policy Details:
    • Name: Enter a descriptive name for your vGPU policy. Ideally, match it to the vGPU profile it references for clarity (e.g., “High Performance vGPU”).
  • vGPU Profile: Select the vGPU profile that defines the type and capabilities of the vGPU to be assigned.
  • Provider VDC Scope : Choose the PVDC has access to the poloicy.
  • Placement: Choose No for placement flexibility. You can assign this policy to VMs and let VCD determine optimal placement based on available resources.
  • Sizing: Select No for sizing flexibility. You can configure VM CPU, memory, and storage independently during VM deployment.
  • Finalize the Policy:
    • Select the Organization VDC where you want this policy to be available.
    • Review the policy details on the “Ready to Complete” screen and click Finish to create the vGPU policy.

Congratulations! You’ve successfully created a vGPU policy in VCD. Now, when deploying VMs in the chosen Organization VDC, you can assign this policy to provide the necessary vGPU power for your graphics-intensive workloads.

Additional Considerations:

  • You can create multiple vGPU policies with different vGPU profiles to cater to varying VM requirements.
  • For more granular control, explore the options for placement and sizing policies within VCD, allowing you to define specific placement rules and resource allocation for vGPU-enabled VMs.

By leveraging vGPU policies, you can efficiently manage and allocate vGPU resources within your VCD environment, empowering your tenants with the graphics processing capabilities they need for their demanding workloads.

VMware Cloud Director Service Crashes During CSE Communication with VCD

Added , By with No Comments

Introduction

In the sphere of cloud management, ensuring uninterrupted service is of paramount importance. However, challenges can emerge, affecting the smooth operation of services. Recently, a noteworthy issue surfaced with a customer – the ‘VMware Cloud Director service crashing when Container Service Extension communicates with VCD.’ This article delves into the symptoms, causes, and, most crucially, the solution to address this challenge.

It’s important to note that the workaround provided here is not an official recommendation from VMware. It should be applied at your discretion. We anticipate that VMware will release an official KB addressing this issue in the near future. The product versions under discussion in this article are VCD 10.4.2 and CSE 4.1.0.

Symptoms:

  • The VCD service crashes on VCD cells when traffic from CSE servers is permitted.
  • The count of ‘BEHAVIOR_INVOCATION’ operation in VCD DB is quite high (more than 10000).
vcloud=# select count(*) from jobs where operation = 'BEHAVIOR_INVOCATION';
 count
--------
 385151
(1 row)
  • In the logs, you may find the following events added in cell.log:
Successfully verified transfer spooling area: VfsFile[fileObject=file:///opt/vmware/vcloud-director/data/transfer] 
Cell startup completed in 1m 39s
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /opt/vmware/vcloud-director/logs/java_pid14129.hprof ...
Dump file is incomplete: No space left on device
log4j:ERROR Failed to flush writer,
java.io.IOException: No space left on device
at java.base/java.io.FileOutputStream.writeBytes(Native Method)

Cause:

The root cause of this issue lies in VCD generating memory heap dumps due to an ‘OutOfMemoryError’ due to this, in turn, leads to the storage space being exhausted and ultimately results in the VCD service crashing.

Resolution:

The good news is that the VMware has identified this as a bug within VCD and plans to address it in the upcoming update releases of VCD. While we eagerly await this update, the team has suggested a workaround in case you encounter this issue:

  1. SSH into each VCD cell.
  2. Check the “/opt/vmware/vcloud-director/logs” directory for java heap dump files (.hprof) on each cell.
cd /opt/vmware/vcloud-director/logs
  1. Remove the files with the “.hprof” extension.
[ /opt/vmware/vcloud-director/logs ]# rm java_xxxxx.hprof
  1. Connect to the VCD database:
   sudo -i -u postgres psql vcloud
  1. Delete records of the operations ‘BEHAVIOR_INVOCATION’ from the ‘jobs’ table:
   vcloud=# delete from jobs where operation = 'BEHAVIOR_INVOCATION';
  1. Perform a service restart on all the VCD cells serially:
   service vmware-vcd restart

By following these steps, you can mitigate the issue and keep your VCD service running smoothly until the official bug fix is released in VCD.

Unable to connect VMware Cloud director 10.4 through PowerCLI

Added , By with No Comments

While attempting to connect to VMware Cloud Director 10.4 using PowerCLI, I encountered the error message “The server returned the following: NotAcceptable: ”.”

 PS C:\Users\sreejesh> Connect-CIServer -Server vcd.sreejesh.lab -Credential (Get-Credential)
Connect-CIServer : 3/27/2023 8:01:29 AM	Connect-CIServer		Unable to connect to vCloud Server 'https://vcd.sreejesh.lab:443/api/'. The server returned the following: NotAcceptable: ''.	
At line:1 char:1
+ Connect-CIServer -Server https://vcd.sreejesh.lab  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Connect-CIServer], CIException
    + FullyQualifiedErrorId : Cloud_ConnectivityServiceImpl_ConnectCloudServer_ConnectError,VMware.VimAutomation.Cloud.Commands.Cmdlets.ConnectCIServer

This issue is a known limitation with PowerCLI versions prior to 13.0. The error occurs because PowerCLI versions earlier than 13.0.0 do not support VMware Cloud Director API versions greater than 33.0. To resolve this issue, the solution is to install the latest version of PowerCLI, version 13.0. If encountering this issue, first confirm that the current PowerCLI version is less than 13.0, and if so, uninstall and reinstall with the latest version to resolve the issue.

 
PS C:\Users\sreejesh> Get-PowerCLIVersion
WARNING: The cmdlet "Get-PowerCLIVersion" is deprecated. Please use the 'Get-Module' cmdlet instead.

PowerCLI Version
----------------
   VMware.PowerCLI 12.7.0 build 20091289
---------------
Component Versions
---------------
   VMware Common PowerCLI Component 12.7 build 20067789
   VMware Cis Core PowerCLI Component PowerCLI Component 12.6 build 19601368
   VMware VimAutomation VICore Commands PowerCLI Component PowerCLI Component 12.7 build 20091293
   VMware VimAutomation Storage PowerCLI Component PowerCLI Component 12.7 build 20091292
   VMware VimAutomation Vds Commands PowerCLI Component PowerCLI Component 12.7 build 20091295
   VMware VimAutomation Cloud PowerCLI Component PowerCLI Component 12.0 build 15940183 



PS C:\Users\sreejesh> Remove-Module -Name VMware.PowerCLI -Force
PS C:\Users\sreejesh> Install-Module -Name VMware.PowerCLI 


PS C:\Users\sreejesh> Get-PowerCLIVersion

PowerCLI Version
----------------
   VMware.VimAutomation.Core 13.0.0 build 20797821
---------------
Component Versions
---------------
   VMware Common PowerCLI Component 13.0 build 20797081
   VMware Cis Core PowerCLI Component PowerCLI Component 13.0 build 20797636
   VMware VimAutomation VICore Commands PowerCLI Component PowerCLI Component 13.0 build 20797821

Enable ssh login in CSE Ubuntu K8s nodes

Added , By with No Comments
  • Login to VMware Cloud Director tenant/provider portal.
  • Access console of the k8s master node.
  • Login as root. You can find the root user password in Guest customization section of the VM in VMware Cloud Director.
  • Open /etc/sshd/sshd_config
  • In the file, find the PasswordAuthentication line and make sure it ends with yes.
  • Save the file and exit.
  • Restart the SSH service by typing the following command:
sudo systemctl restart sshd
  • Its good to create a user for ssh login rather than enabling root login over ssh.
#useradd kube
#passwd kube
  • You can ssh to the master node with kube user and do ‘su -‘.

How to validate TKGm Cluster?

Added , By with No Comments
Reply

Please find the steps to validate a TKGm cluster deployed through VMware Container Service Extension.

Step 1 : Download kubeconfig file

  • Download the Kubeconfig file to a windows machine which has access to the Native Kubernetes cluster.
  • Create folder .kube under $HOME.

$HOME\.kube

  • Copy the configfile dowloaded to .kube folder.
  • Rename the file to ‘config’ without any extensions.

Step 2 : Download kubectl

  • Download Kubectl for Windows from 
https://dl.k8s.io/release/v1.22.0/bin/windows/amd64/kubectl.exe
  • Create folder $HOME\kubectl and copy kubectl.ext to the folder. Add the folder to the ‘Path’ User variable in Environment Variables.

Run kubectl

Step 3: Run a ‘hello world’ application in the cluster.

Follow the steps from following article to deploy a Hello World applicaiton in the K8S cluster created.

Exposing an External IP Address to Access an Application in a Cluster | Kubernetes

Note: In the following command use NodePort instead of LoadBalancer

kubectl expose deployment hello-world --type=LoadBalancer --name=my-service

How to create NSX-T Routed network in VCD for Tanzu Kubernetes Grid (TKG) clusters?

Added , By with No Comments

Please find the steps for configuring the Network in VCD for deploying TKG clusters.

Add the public IP to the Static IP Pool of T0 GW

  • Login to VCD Provider portal.
  • Navigate to Resources > Cloud Resources > Tier-0 Gateways.
  • Select the T0 Gateway.
  • Select ‘Network Specification’
  • Edit
  • Add the Public IP(s) to the ‘Static IP Pools’

Create Edge Gateway (T1 Router)

  • Login to VCD Provider portal.
  • Navigate to Resources > Cloud Resources. > Edge Gateways
  • Select New
  • Select the Org VDC and click Next
  • Provide a name for the Edge.
  • Select the appropriate T0 Gateway
  • Choose the appropriate Edge Cluster option for your environment.
  • Assign the Public IP for SNAT as Primary IP
  • Cleck Next review the settings and click Finish.

Create Organization Network

  • From provider portal select the Test organization.
  • Navigate to Networking > Networks.
  • Click New
  • Select Org VDC
  • Select Network Type ‘Routed‘
  • Select the Edge Gateway (T1)
  • Provide the Name and Gateway CIDR
  • Provide the DNS server accessible from the Org Network created. The DNS server should be able to resolve the FQDNS in the public domain/internet.
  • Click Next, review the settings and click on Finish.

Create SNAT

  • From provider portal select the Test organization.
  • Navigate to Networking > Edge Gateways
  • Select the Edge Gateway (T1)
  • Navigate to Services > NAT
  • Click New
  • Provide the details as mentioned in the screenshot.

Modify default Firewall rule

  • From provider portal select the Test organization.
  • Navigate to Networking > Edge Gateways
  • Select the Edge Gateway (T1)
  • Navigate to Services > Firewall
  • Select ‘Edit Rules’
  • Select the ‘default_rule’
  • Edit
  • Select Allow as Action.

How to run VMware Container Service Extension (CSE) as Linux Service?

Added , By with No Comments

After installing CSE please follow the steps below to run it as a service.

Create cse.sh file

Create cse.service file. You can copy the following code or create new one based on following link.
container-service-extension/cse.sh at master · vmware/container-service-extension (github.com)

# vi /opt/vmware/cse/cse.sh
#!/usr/bin/env bash
export CSE_CONFIG=/opt/vmware/cse/encrypted-config.yaml
export CSE_CONFIG_PASSWORD=<passwd>
cse run

Copy encrypted-config.yaml to /opt/vmware/cse directory.

Change the file permission

chmod +x /opt/vmware/cse/cse.sh

Create cse.service file

Create cse.service file. You can copy the following code or create new one based on following link.
container-service-extension/cse.service at master · vmware/container-service-extension (github.com)

vi /etc/systemd/system/cse.service
[Unit]
Description=Container Service Extension for VMware Cloud Director

[Service]
ExecStart=/opt/vmware/cse/cse.sh
User=root
WorkingDirectory=/opt/vmware/cse
Type=simple
Restart=always

[Install]
WantedBy=default.target

Enable and start the service

# systemctl enable cse.service
# systemctl start cse.service

Check the service status

# systemctl status cse.service
  cse.service - Container Service Extension for VMware Cloud Director
   Loaded: loaded (/etc/systemd/system/cse.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-11-24 14:43:56 +01; 1min 9s ago
 Main PID: 770 (bash)
   CGroup: /system.slice/cse.service
           ├─770 bash /opt/vmware/cse/cse.sh
           └─775 /usr/local/bin/python3.7 /usr/local/bin/cse run

Nov 24 14:44:06 cse01.lab.com cse.sh[770]: Validating CSE installation according to config file
Nov 24 14:44:06  cse.sh[770]: MQTT extension and API filters found
Nov 24 14:44:06 cse01.lab.com cse.sh[770]: Found catalog 'cse-site1-k8s'
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: CSE installation is valid
Nov 24 14:44:06 cse01.lab.com cse.sh[770]: Started thread 'MessageConsumer' (140229531580160)
Nov 24 14:44:06 cse01.lab.com l cse.sh[770]: Started thread 'ConsumerWatchdog' (140229523187456)
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: Container Service Extension for vCloud Director
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: Server running using config file: /opt/vmware/cse/encrypted-config.yaml
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: Log files: /root/.cse-logs/cse-server-info.log, /root/.cse-logs/cse-server-debug.log
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: waiting for requests (ctrl+c to close)