Build & expand

Private AI Foundation — Connected-Site Runbook

The step-by-step companion to 33-private-ai-foundation.md for a site with internet access. A connected build pulls most artifacts straight from Broadcom and NVIDIA, so there is little to stage: the work is the order, who does it in which console, the tokens, and the settings that are easy to miss. For a disconnected site use 36-paif-dark-site-walkthrough.md; the install and activation of Private AI Services in §7 and §8 is shared by both.

Evidence level: from the VMware Private AI Foundation with NVIDIA 9.1 PDF (read 2026-10-30) and KB 388470. Not run on a build in this repo yet. Where a step is general practice rather than Broadcom’s text, it says so.

Prerequisite for everything: the GPU hosts and the GPU workload domain with a Supervisor (32-gpu-hosts.md).


The flow

Connected build flow: eight steps across four roles and consoles, left to right: tokens and keys, licence service, VM classes in the vSphere Client, region and quota in Provider Management, library, namespace class, project and namespace and the Quickstart in the organization, install Private AI Services in the vSphere Client, activate it in a namespace

Click the picture to open it full size.

Each step has an owner and a console. A connected build has no air gap to cross, so the hand-offs are between people and consoles instead: the VI admin creates what the provider admin then shares, and the provider admin shares what the org admin then uses.

#TaskRoleConsoleDone when
1Collect tokens and keysPlatform + NVIDIA licence ownerSupport Portal, NGCAll four items in hand
2Licence service and client tokenPlatformNVIDIA Licensing PortalClient configuration token generated
3VM classesVI adminvSphere Client (management vCenter, and the workload vCenter for the DirectPath profile)Reserved VM classes exist per zone
4Region and region quotaProvider adminVCF Automation Provider ManagementThe organization has a quota in the GPU zone
5Library, namespace class, project, namespaceOrg adminVCF Automation organization (+ CLI for reserved classes)A namespace exists in a project
6QuickstartOrg adminVCF Automation organizationFour catalog items visible
7Install Private AI ServicesVI adminvSphere ClientService shows as configured
8Activate Private AI ServicesOrg admin or VI adminVCF Automation or vSphere ClientModel endpoints can be created

Steps 1 and 2 can start on day one. Step 3 needs the Supervisor. Steps 7 and 8 are only for Private AI Services; the Quickstart (6) is for deep learning VMs and VKS clusters. You can do either path, or both.


1. Collect tokens and keys

ItemWhere fromNeeded for
NVIDIA client configuration tokenYour NVIDIA License Service instance (33 §2)Quickstart, deep learning VMs, GPU Operator
NGC API keyNVIDIA NGC (API keys)Pulling NVIDIA AI Enterprise containers
Broadcom registry tokenSupport Portal → My Downloads → Registry Tokens → Generate Token → copy the tokenPrivate AI Services from pais-docker.packages.broadcom.com; on 9.1.1 dlvm.packages.broadcom.com
Deep learning VM library URLhttps://dl.broadcom.com/<download_token>/PROD/COMP/DLVM/<version>/lib.json (no authentication)Content library, step 5

Passwords you type into the Quickstart (a private registry, a proxy) are saved as secrets in VCF Automation (PAIF 9.1).


2. NVIDIA licence service

For vGPU only (passthrough needs no NVIDIA licence). On the NVIDIA Licensing Portal, configure a License Service instance and generate the client configuration token (NVIDIA License System docs). Keep the .tok content somewhere the org admin can reach it: the Quickstart asks you to paste its contents.

Behind a proxy: the Quickstart’s Cloud option supports only an unauthenticated HTTP proxy for vGPU licence registration (PAIF 9.1). Plan an exception if your proxy requires a login.


3. VM classes

Where: the vSphere Client, on the management vCenter (https://<management_vcenter_fqdn>/ui), except step 1 which is on the workload domain vCenter. Background and rationale: 33 §3.

  1. DirectPath Profile (only for classes with GPU reservation), on the workload domain vCenter: Policies and Profiles → DirectPath Profiles → Create. Select the workload domain’s vCenter and the vGPU or passthrough device, then give it a name.

  2. Create the VM classes: Private AI Foundation → Set Up Environment → Create VM classes with NVIDIA vGPUs → Start; select the GPU workload domain; then:

    SettingWhat to enter
    CompatibilityESX 9.0 and later
    CPU, memoryThe class size
    Reservation and limitGPU-reserved classes: the same non-zero value for both, for CPU and for memory
    MemoryReserve all guest memory: AI/ML workloads need full memory reservation
    PCI deviceThe NVIDIA vGPU profile or the passthrough device
    Network adapter (optional)Enhanced DirectPath I/O on an accelerated NIC, for GPUDirect RDMA
    Passthrough GPUs mapping > 16 GBpciPassthru.use64bitMMIO = true, and pciPassthru.use64bitMMIOSizeGB sized as below

    Sizing pciPassthru.use64bitMMIOSizeGB (PAIF 9.1): number of passthrough GPUs in the class × the frame buffer of one GPU; if the total is a power of two, take the next power of two; if it lies between two, round up to the next, then once more. Example: two A100 40 GB → 80 GB → 128 → 256.

  3. Reserve capacity per zone: Supervisor Management → the Supervisor → Zones → ⋮ → Add Reserved VM Class; pick a vGPU or passthrough class, set the number of instances to reserve, Add Reservation; repeat per class. VCF Automation hands these instances out to organizations later.

  4. Leave failover capacity. Broadcom: in a 16-host cluster keep one host as dedicated failover and let the rest carry zonal GPU reservations without exceeding the capacity of 14 hosts.

  5. CPU-only models (Private AI Services on llama.cpp): make a VM class with enough CPU; no GPU settings.

Check: the classes appear under the Supervisor’s zones; the reserved counts match what you intend to give the organization.


4. VCF Automation region and quota

Where: Provider Management, as provider admin. Before starting, confirm (PAIF 9.1): the Supervisor’s NSX Manager is registered in VCF Automation, the zones are Ready (otherwise capacity is shown wrongly), an All Apps organization exists, and VPC networking for it has a public IP range, a transit gateway IP range and a VPC private IP range.

Create the region (Infrastructure → Regions → Create Region):

  1. Name (for example paif-region).
  2. The NSX Manager that integrates with the vCenters you use: one NSX Manager per region.
  3. One or more Supervisors.
  4. One or more storage classes: they must exist with the same names on every vCenter in the region.
  5. Submit.

Create the region quota (Infrastructure → Organizations → the organization → Region Quota → New):

  1. Select the region; under Region Capacity choose Share a portion of the capacity with this organization.
  2. To pick Supervisors, turn off Add all available supervisors and select them.
  3. Add the zones and set CPU and memory limits and reservations: only finite values.
  4. Assign and Continue; select the non-reserved VM classes and storage classes (you can edit each storage limit); Assign and Continue.
  5. Optional: add reserved VM classes with counts (vGPU or passthrough).
  6. Continue.

Rules that bite (PAIF 9.1):

  • CPU and memory limits can be over-provisioned; reservations cannot.
  • A reserved VM class’s count must fit the capacity of that zone on the Supervisor, not the aggregated capacity of the zone.
  • You cannot give a guaranteed VM class to a quota with zero reservation, or one where reservations are not given to every zone.
  • If a namespace is created with GPU reservations, a non-zero memory reservation must be at least 2 GB (guest memory overhead).
  • Group GPUs of one type per cluster: a namespace is created on one cluster, so its reserved classes must be the same GPU type on the same cluster.

VCF Automation refreshes its inventory every 15 minutes. After changing a reserved class in vCenter, use Refresh on the vCenter in Provider Management instead of waiting.


5. VCF Automation organization content

Where: the organization UI, as organization administrator.

5.1 Content library for deep learning VMs (Build and Deploy → Content Hub → Content Libraries → Create Content Library):

  1. Name it; optionally turn on the toggle that assigns it to all current and future namespaces.
  2. Turn on Subscribe to an external library, Next, and enter https://dl.broadcom.com/<download_token>/PROD/COMP/DLVM/<version>/lib.json (no authentication). Next.
  3. Select one or more regions and a storage class for each; Next; review; Confirm.

5.2 Namespace class (Manage & Govern → Namespace Class → New): name, CPU and memory limits and reservations, at least one VM class and one storage class, and the library from 5.1; Create. A namespace class holds either reserved or non-reserved GPU classes, not both: mixed, VCF Automation treats it as reserved and requests with the non-reserved class fail (PAIF 9.1).

Reserved GPU classes: the PDF’s route for a namespace class with reserved GPU VM classes is the CLI or API (the UI example lists non-GPU and unbound GPU classes). On a machine with the VCF Consumption CLI and kubectl:

vcf context create cci -e <vcf_automation_endpoint> -t cci \
  --api-token ${vcf_automation_organization_api_token} \
  --tenant-name <org-name> --ca-certificate /tmp/vcfa-ca-cert.pem
vcf context use cci

kubectl get regions
kubectl get zones --field-selector spec.regionName=paif-region
kubectl get regionvirtualmachineclasssummaries --field-selector spec.regionName=paif-region
kubectl get regionvirtualmachineclassquotas    --field-selector spec.regionName=paif-region
kubectl get regionstorageclassquotas           --field-selector spec.regionName=paif-region
kubectl get vpcs                               --field-selector spec.regionName=paif-region

Create the class, then its configuration (names, zones and values are the PDF’s examples; adapt them):

apiVersion: infrastructure.cci.vmware.com/v1alpha2
kind: SupervisorNamespaceClass
metadata:
  name: medium-gpu-workloads
spec:
  description: supervisor namespace class
---
apiVersion: infrastructure.cci.vmware.com/v1alpha2
kind: SupervisorNamespaceClassConfig
metadata:
  name: medium-gpu-workloads
spec:
  contentSources:
    - name: "paif-content-library"
      type: ContentLibrary
  storageClasses:
    - name: 'vSAN Default Storage Policy'
      limit: 2000Gi
  vmClasses:
    - name: 'best-effort-large'
    - name: 'vgpu-1xa100-40c'
  zones:
    - name: domain-c10
      cpuLimit: 40G
      cpuReservation: 10G
      memoryLimit: 50Gi
      memoryReservation: 10Gi
      vmClassReservations:
        - count: 1
          vmClassName: vgpu-1xa100-40c

Apply with kubectl apply -f <file>. A * for storageClasses or vmClasses lets users override the limits or see every class.

5.3 Project, then namespace (Manage & Govern → Projects → New; then the project → Namespaces → New Namespace): name; the namespace class from 5.2; the region; at least one zone; a networking type. Optionally turn on the Override reserved VM classes toggle to assign reserved classes per zone (the count cannot exceed Max Available) and the Override toggle for resource limits (only if the zones are not defined in the class).

Check: the namespace is created; in vCenter a resource pool exists for it on the clusters of the chosen zones.


6. Run the Quickstart (deep learning VMs and VKS)

Skip if you only use Private AI Services. As organization administrator: Overview → Private AI Foundation Quickstart → Start. You prepare both the deep learning VM and the VKS side even if you plan to use one.

Wizard pageWhat to enter
Where to provisionThe project and a namespace with vGPU-enabled VM classes and the Private AI images
NVIDIA modevGPU: paste the client configuration token. Fixed/Dynamic DirectPath: whole-GPU passthrough, no NVIDIA AI Enterprise licence. Only asked if the namespace has both
ConnectivityCloud: drivers from the NVIDIA Licensing Portal, containers from NGC. Optional proxy (unauthenticated HTTP only for licence registration); add the external IP block, private TGW IP block and Private AI VPC CIDR (from Manage & Govern → Networking → IP Management → IP Address Blocks) to Do not use proxy for, comma-separated, so Kubernetes traffic bypasses it
Names and contentA suffix for the catalog item names; the Kubernetes version (one of the three newest supported); the deep learning VM image; the storage class
VM classesAt least one GPU class and one non-GPU class. GPU classes: reserved or non-reserved, not both, filtered by the mode. The non-GPU class runs the Kubernetes control planes. UVM-enabled classes are required for AI workstations with Triton, vGPU only
Data Services Manager (optional)Toggle on to create the DSM Database item: replica mode (Single Server, or Single vSphere Cluster with three nodes), an infrastructure policy, and optionally a default storage policy and VM class. DSM VM classes differ from Supervisor VM classes

Review, save your settings somewhere, Finish. Result: AI Workstation, Triton Inference Server, AI Kubernetes Cluster, DSM Database. Run it again to add another Supervisor, a new token, a new driver URL, a new image, other VM classes or a different proxy; each run adds items next to the old ones.

After the Quickstart:

  • 9.1.1: replace tkg-gpu-console in the AI Kubernetes Cluster blueprint (PAIF 9.1, KB 388470). Needs a public project in the platform Harbor and a machine with Docker:

    echo <registry_token> | docker login dlvm.packages.broadcom.com \
      --username <support-portal-login> --password-stdin
    docker pull dlvm.packages.broadcom.com/dlvm/ga/9.1.1.0/deep-learning-container:9.1.1.0
    docker tag  dlvm.packages.broadcom.com/dlvm/ga/9.1.1.0/deep-learning-container:9.1.1.0 \
                <harbor-fqdn>/<public-project>/deep-learning-container:9.1.1.0
    docker push <harbor-fqdn>/<public-project>/deep-learning-container:9.1.1.0

    Then edit the blueprint (Build & Deploy → Content Hub → Blueprint Design → Blueprints): change image: paif.packages.broadcom.com/tkg-gpu-console:9.0.0.0 to image: <harbor-fqdn>/<public-project>/deep-learning-container:9.1.1.0, save and publish.

  • 9.0 / 9.1.0: GPU Operator v25.3.1 in the blueprint is end-of-life; update to v26.3.1 (KB 439984).

  • Check the catalog items are visible to the project members, and watch the first provisioning.


7. Install Private AI Services (the Supervisor Service)

Shared with the dark-site build. Where: the vSphere Client, as VI admin. Prerequisites in 33 §9: Data Services Manager, an OIDC provider, Harbor as the Supervisor’s default container registry. Install source (3.0):

Where the package isYAML to downloadWhen
Software depot, through a regional Harborpais-3.0.0.ymlVCF 9.1.1, connected
pais-docker.packages.broadcom.compais-3.0.0-external.yaml9.1.0 or 9.1.1, connected or disconnected

Steps (PAIF 9.1, Install Private AI Services on the Supervisors of the Workload Domain):

  1. Registry credentials on the Supervisor (only when installing from pais-docker.packages.broadcom.com). In the management vCenter: Supervisor Management → the Supervisor → Configure → Supervisor → Container Registries → Add Registry:

    FieldConnectedDisconnected
    Registry namepais-reg-credentialssame
    Host URLpais-docker.packages.broadcom.comyour Harbor URL
    UsernameSupport Portal login e-mailHarbor username
    PasswordBroadcom registry tokenHarbor password (the PDF lists one value, “your OCI registry access token”)

    When installing 3.0 from the depot, skip this: the regional Harbor connected to the depot must hold the package in its vcf project and be the Supervisor’s default container registry.

  2. Private AI Foundation → Set Up Environment → Start → Install the Supervisor Service for Private AI Services → Start.

  3. Select the workload domain, upload the YAML, then choose the service version and the Supervisor. Connected: leave YAML Service Config empty. Disconnected: paste the content of the yaml-svc-cfg.yaml produced by the artifact mirroring tool, and use the pais YAML the tool created (36 phase 7).

Manage the service later from Supervisor Management → Services → Private AI Services. Private AI Services can be updated without updating VCF.


8. Activate Private AI Services in a namespace

The service is installed but inactive in every namespace: you see the Private AI entry, but cannot create model endpoints yet (PAIF 9.1).

Before you start (collect these first):

  • A VM class and storage policy in the namespace for the control plane of the VKS cluster that runs model endpoints (no GPU needed there).

  • An FQDN for the instance if your organization needs one (map it to the IP after activation).

  • For knowledge bases, MCP tools or agents: a PostgreSQL database with the pgvector extension (Data Services Manager creates it with the extension; third-party works if pgvector is installed) and its CA certificate.

  • OIDC details. External provider: issuer URL, client ID, the groups allowed (or an empty list), scope and claim names if not groups, and the issuer certificate. VCF Automation as provider: the client ID from the OIDC provider proxy in Provider Management and the VCF Automation issuer certificate:

    openssl s_client -connect <vcf_automation_fqdn>:443 -showcerts </dev/null 2>/dev/null \
      | awk 'BEGIN{c=0;p=0} /-----BEGIN CERTIFICATE-----/{c++;if(c>1)p=1} p{print} /-----END CERTIFICATE-----/{p=0}'
  • The provider must support the Authorization Code flow with PKCE and OIDC Discovery; if it supports the offline_access scope, turn it on (refresh tokens). Not VCF SSO.

Run the wizard:

  1. Open it. VCF Automation: All Apps organization → Build & Deploy → the namespace → Services → Private AI → Configure Service. vSphere Client: Supervisor Management → Namespaces → the namespace → Resources → Private AI → Configure Service (needs the Local Consumption Interface installed on the Supervisor).
  2. Service Resources: choose the full set of capabilities (adds MCP server integration, Data Indexing and Retrieval, Agent Builder, and the database) or Model Runtime only (optionally with metrics). Pick the default storage policy for the management components, and the VM class (and optional storage policy) for the VKS control plane. Worker nodes are created per endpoint, with resources set at endpoint deployment.
  3. Database (full path only): PostgreSQL Cluster managed by Data Services Manager if it was created in VCF Automation in the same namespace; otherwise Bring your own PostgreSQL database with its connection details and CA certificate.
  4. User Access: add an OIDC provider (external: issuer URL, client ID, certificate, optional authorized groups; VCF Automation: issuer https://<vcf_automation_fqdn>/oidc, client ID, allowed organizations, roles, groups, TLS verification mode). Local accounts are for proof of concept and cross-organization model sharing. With VCF Automation as provider, add its issuer certificate as a CA trust bundle to the instance after activation, in every namespace that uses it.
  5. GPU Driver: Use CPUs Only (llama.cpp), or the NVIDIA vGPU guest driver with the client configuration token in vGPU Driver License Token Config Map and GPU Operator overrides (a values template per the release notes’ supported version). driver.version in the overrides is not honoured; for a different GPU Operator or driver version use the CLI after activation. Leave the custom OCI registry fields empty: the components are already on the Supervisor.
  6. Finish the wizard, then check the instance reaches ready.

Check: you can create a model endpoint in the namespace, and the OIDC login works from a browser that trusts the issuer.


Troubleshooting

SymptomCauseFixSource
Capacity shows wrongly in the quotaZones not Ready in Provider ManagementFix the zone status firstPAIF 9.1
A new or changed reserved VM class is not visibleInventory refreshes every 15 minRefresh on the vCenter in Provider ManagementPAIF 9.1
Requests with a non-reserved GPU class failThe namespace class also holds a reserved GPU classKeep reserved and non-reserved in separate classesPAIF 9.1
Cannot add a guaranteed VM class to the quotaZero reservation, or not every zone has reservationsGive CPU and memory reservations in all zonesPAIF 9.1
Namespace with GPU reservation is rejectedMemory reservation under 2 GB, or counts exceed the zoneReserve ≥ 2 GB; fit the zone’s capacityPAIF 9.1
kubectl/vcf says no regionsWrong context, or the quota is missingvcf context use cci; check the quotaPAIF 9.1
Quickstart licence registration fails behind a proxyProxy needs authenticationUse an unauthenticated HTTP proxy, or no proxyPAIF 9.1
AI Kubernetes Cluster pulls an EOL GPU OperatorBlueprint default v25.3.1 (9.0/9.1.0)Update to v26.3.1KB 439984
deep-learning-container:9.1.1.0 not pulledNot logged in to dlvm.packages.broadcom.comdocker login firstKB 388470
Private AI Services cannot reach its OIDC providerTried VCF SSO / Identity BrokerUse an external OIDC provider or VCF AutomationPAIF 9.1
Service cannot pull from HarborHarbor CA not trusted by the SupervisorAdd the CA under Container RegistriesLam

References

VMware Private AI Foundation with NVIDIA 9.1 (PDF) · KB 388470 · KB 439984 · NVIDIA License System · NGC API keys · William Lam — PAIS AMT for air-gapped · 33-private-ai-foundation.md · 36-paif-dark-site-walkthrough.md · 17-vcfa-tenant-config.md

ITQ – What's Next. VCF 9.1 deployment-planning and operations helper material. © 2026 Paul van Dieen. hollebollevsan.nl