Verification

Test Plan

The Deployment Plan says what to build and the Tracker says how far you got. This says how you prove it works. Every case names the component under test — the product you log in to, and whose owner a failure goes to — plus the epic and story it verifies, so a green test plan maps one-to-one onto a finished deployment plan. The scope follows what you set in the export tool — adjust it there and this page picks it up. Results are remembered by this browser; use Save results to keep the file with the engagement (it is customer data: store it with the customer's files, not in a repo). Everything runs in your browser.

Scope: (loading)

Adjust scope in the export tool
How to run this
  1. Set the scope once. Build it in the Deployment Plan Export tool — connectivity, storage, stretch, Day-2 components, workload domains. This page filters to the cases that scope actually needs, so you are never reading past tests for things you did not deploy.
  2. Work a phase at a time, in order. Each phase has a gate. Do not start the next phase while the current one still has an open F1 — the whole point of the ordering is that a broken foundation fails the later tests in confusing ways.
  3. Run TP-0 before bring-up. It is the readiness gate: DNS both ways, NTP, MTU, storage, depot, firewall. Most failed deployments are a TP-0 case nobody ran.
  4. For each case: follow the numbered steps, compare against Expected, then click P, F1, F2 or NA. The date stamps itself. Click the same button again to clear the case.
  5. Always fill the actual-result box. Put the measured value, the command output, or a pointer to your evidence file in it — for a pass. For an F2, put the owner and the agreed date. For an NA, put the reason. A tick with an empty box is not a pass, and the report will list it as incomplete.
  6. Hand over. Download the Excel workbook as the artefact the customer fills in and keeps, the Report as the write-up, and keep the Results file with the engagement so a colleague can carry on. Once you are filling the workbook in Excel, Excel is the source of truth — it does not sync back.

Filters help on a long run: Critical cases only gives you the go/no-go set, and Hide executed leaves just what is still outstanding.

P pass · F1 critical fail (blocks the gate) · F2 non-critical fail · NA not applicable · Automatable can be verified by tooling rather than by hand

Four exports, four jobs. Excel is the one to take on site: a workbook laid out the way field verification workbooks are — a sheet per phase, Status as a dropdown, and a live percentage block and summary that recalculate as it is filled in. Report is the write-up: a verdict, per-phase results, the open critical failures and carried actions — no test steps. CSV is the flat import path for a test-management tool. Runbook is the procedure in full. Save results keeps the JSON with the engagement (it is customer data — store it with the customer's files, not in a repo); it carries the scope too, so loading it restores the exact plan.

TP-0 — Readiness E1–E4 0/13

Gate: The pre-bring-up go/no-go. Everything the plan called for has actually been built.

  • TP-001 ToR fabric Management VLANs trunked to every host uplink at the planned MTU Critical E1 · Story 1.1 / 4.2
    1. On the ToRs, list the allowed VLANs per host-facing port and compare against your VLAN table. Cisco NX-OS: show interface trunk. Arista EOS: show interfaces trunk. Dell OS10: show interface status.
    2. Confirm the same VLAN set is on BOTH uplinks of every host — a VLAN trunked on one uplink only survives a casual test and fails at the first uplink outage.
    3. On a host, check the uplinks are up at the expected speed: esxcli network nic list
    4. Check the switch MTU as configured: esxcli network vswitch dvs vmware list | grep -i mtu (or, for a standard switch, esxcli network vswitch standard list).
    5. Now PROVE the MTU instead of reading it. From a host, ping a peer on each jumbo VLAN (vSAN, vMotion, host overlay) with do-not-fragment set at 8972 bytes payload — 9000 minus the 28-byte IP+ICMP header: vmkping -I vmk1 -d -s 8972 <peer-vmk-ip>
    6. Repeat to a peer in a DIFFERENT rack, so the test crosses the inter-switch link and any L3 hop rather than staying inside one ToR.
    7. List the VMkernel interfaces if you need the right vmk number and its IP: esxcli network ip interface ipv4 get

    Expected: Every planned VLAN is trunked on both uplinks of every host, and an unfragmented 8972-byte ping succeeds across each jumbo VLAN — including between racks.

    An MTU mismatch that only appears on the inter-rack path is the classic cause of a bring-up that stalls at cluster creation.

  • TP-002 IP plan Subnets and IP reservations do not overlap E1 · Story 1.1
    1. List every planned subnet: management, vMotion, vSAN/storage, host overlay, edge/uplink or external, the fleet component ranges, the VCF services-runtime cluster CIDR, and on Distributed connectivity the private transit-gateway block.
    2. Lay the ranges side by side — sorting them by network address makes an overlap obvious to the eye, which is enough for the handful of subnets a deployment uses. A subnet calculator (e.g. an online CIDR tool, offline) resolves any case you are unsure of.
    3. Ask the network team to confirm none of the ranges is already routed elsewhere — an overlap with something you cannot see from the build network is the one a desk check cannot catch.
    4. Confirm the static IP carve-outs are reserved in the IPAM system so nobody is handed one mid-build.

    Expected: No subnet overlaps another planned subnet or an existing routed network, and every static reservation is recorded in IPAM.

    The services-runtime CIDR and the transit-gateway block are internal to VCF but still have to be unique — they are the two most commonly forgotten.

  • TP-003 ToR fabric ToR BGP fabric is configured and ready to peer Critical E1 · Story 1.2 / 4.2
    1. On the ToRs, confirm the local AS and the neighbour statements for the planned edge uplink IPs. Cisco NX-OS: show running-config bgp then show bgp sessions. Arista EOS: show running-config section bgp then show ip bgp summary.
    2. Confirm the remote AS on each neighbour statement matches the Edge AS in your plan, and that BFD is set if the design calls for it: show bfd neighbors
    3. If BGP authentication is in scope, confirm the MD5 password is configured on the ToR side and recorded in the secret store — and that both sides will agree on it.
    4. Confirm the accept/advertise policy: which prefixes the fabric will take from the edges, and which it will advertise to them. show ip prefix-list / show route-map
    5. Confirm the uplink VLANs and their gateway SVIs exist and answer: show ip interface brief on the ToR, then ping each SVI from a host on that VLAN.

    Expected: ToR-side BGP configuration is complete and waiting on the edges: neighbours defined with the correct remote AS, uplink VLANs live, and the accept/advertise policy agreed in writing.

    VCF/NSX only needs the neighbour IP and remote AS — an MD5 password is optional and only required if the fabric enables authentication.

  • TP-006 DNS Forward and reverse DNS resolve for every planned FQDN Critical E1 · Story 1.3 / 4.3
    1. Put every planned FQDN in a text file, one per line: every ESX host, vCenter, SDDC Manager, the NSX Manager VIP and all three nodes, the Edge nodes (Centralized) or VNA appliances (Distributed), the VCF Operations nodes and any external LB VIP, the Cloud Proxy, the License Server, every VCF Management Services FQDN, and any Day-2 appliance FQDNs already known.
    2. Check forward AND reverse from a client on the management network. For each name: Resolve-DnsName <fqdn> -Type A, then feed the returned IP back in: Resolve-DnsName <ip> -Type PTR. The PTR must return the same name you started with.
    3. From Linux or a shell: while read f; do ip=$(dig +short A "$f"); ptr=$(dig +short -x "$ip"); echo "$f -> $ip -> $ptr"; done < fqdns.txt
    4. Spot-check a couple of names from an ESX host too, since it uses its own resolver config: nslookup <fqdn> and esxcli network ip dns server list
    5. Run this from the management network, NOT on the DNS server itself — a record that resolves locally but is not served to the management subnet still fails bring-up.

    Expected: Every planned FQDN resolves forward to its planned IP, and every planned IP resolves back to the same name. No missing PTR records.

    Missing PTR records are the most common single cause of a failed bring-up. Forward-only is not a pass.

  • TP-007 NTP NTP sources are reachable and serving from the management subnet Critical E1 · Story 1.3 / 4.3
    1. From Windows, query each planned source directly and read the offset: w32tm /stripchart /computer:<ntp-server> /samples:5 /dataonly
    2. From Linux or an ESX host: ntpdate -q <ntp-server> — check the reported stratum and offset are sane (stratum well under 16, offset small).
    3. Query every planned source and confirm they agree with each other to within a second. Two sources a minute apart is worse than one source.
    4. Confirm UDP 123 is permitted from the management subnet: Test-NetConnection <ntp-server> -Port 123 -InformationLevel Detailed (note TCP test only proves routing; the w32tm/ntpdate query above is the real check).
    5. On an already-built ESX host, confirm it is actually synced rather than just configured: esxcli system ntp get then esxcli system ntp stats get

    Expected: Every planned NTP source answers from the management subnet and the sources agree with each other.

    Time skew breaks certificate validation and SSO in ways that surface much later as unrelated-looking failures.

  • TP-008 Certificate authority Certificate authority is reachable and issues from the chosen template E1 · Story 1.4
    1. Confirm the CA type matches the plan — Microsoft CA or OpenSSL are both supported fleet CA types. An external CA is CSR-based only: VCF will not import an externally created certificate plus its private key.
    2. Microsoft CA: confirm it is reachable and list the available templates — certutil -config "<ca-host>\<ca-name>" -ping then certutil -template | findstr /i "TemplatePropCommonName"
    3. Generate a throwaway CSR: openssl req -new -newkey rsa:2048 -nodes -keyout test.key -out test.csr -subj "/CN=cert-test.<domain>" -addext "subjectAltName=DNS:cert-test.<domain>"
    4. Submit it against the intended template: certreq -submit -config "<ca-host>\<ca-name>" -attrib "CertificateTemplate:<TemplateName>" test.csr test.cer
    5. Inspect what came back: openssl x509 -in test.cer -noout -text | findstr /i "Signature Algorithm Public-Key Not Before Not After Key Usage DNS"
    6. Check it against fleet requirements: RSA 2048 or better, SHA-256 or better, Key Usage includes Digital Signature and Key Encipherment, Extended Key Usage includes Server Authentication and Client Authentication, and the SAN carried through.
    7. Delete the throwaway certificate and key when done.

    Expected: A test certificate is issued from the intended template and its key usage, key size, and validity match fleet requirements.

    Discovering the template is wrong during the fleet-wide replacement (TP-428) costs a day. It costs ten minutes here.

  • TP-009 Design & documentation Sizing fits the proposed hosts at N-1 E2 · Story 2.2
    1. Run the sizing calculation for the agreed component set on the Sizing calculator tool (/tools/mgmt-sizing/), including any Day-2 fleet and License Hub footprint.
    2. Compare CPU, memory, and usable capacity against the proposed hosts with ONE host removed from the cluster.
    3. For vSAN, size against usable capacity after the chosen failure tolerance — and after site mirroring if the cluster will be stretched, which roughly doubles consumption.

    Expected: The full component set fits within N-1 host capacity, or the host count/spec has been adjusted and re-signed by the architect.

  • TP-010 ESX Hosts are on the compatibility guide, identically specified, and meet the cluster minimum Critical E4 · Story 4.1
    1. Dump the fleet spec in one pass with PowerCLI and eyeball it for outliers: Get-VMHost | Select Name,Model,ProcessorType,NumCpu,CpuTotalMhz,@{n="RAM_GB";e={[math]::Round($_.MemoryTotalGB)}},Version,Build | Sort Name | Format-Table -Auto
    2. Every row should be identical apart from the name. A single mismatched host is accepted at commission and causes imbalance later.
    3. List the storage controllers and NICs to check against the compatibility guide: Get-VMHost | Get-VMHostPciDevice | Where-Object DeviceClassName -match "Serial Attached SCSI|RAID|Ethernet" | Select VMHost,DeviceName,VendorName
    4. Per host, get the driver and firmware for the storage controller: esxcli storage core adapter list then vmkload_mod -s <driver> | grep -i version
    5. Check each model, controller and NIC against the Broadcom compatibility guide for the target release, at the installed driver/firmware level — the model being listed is not enough if the firmware is not.
    6. Confirm the host count meets the cluster minimum for the chosen configuration: (Get-Cluster <name> | Get-VMHost).Count

    Expected: Every host is listed on the compatibility guide at its installed firmware/driver level, all hosts in a cluster are identically specified, and the count meets the minimum.

  • TP-012 vSAN vSAN disks are present, unclaimed, and eligible E4 · Story 4.1
    1. On each host, list which devices are eligible for vSAN and why the ineligible ones are not: vdq -q — look for "State":"Eligible for use by VSAN". Anything reporting Ineligible gives its reason in the same output.
    2. A device that is ineligible because it holds an old partition can be cleared with partedUtil — but confirm it really is spare before wiping anything.
    3. List the devices with their transport and size to confirm they are the ones you intend: esxcli storage core device list | grep -E "Display Name|Size|Is SSD|Device Type"
    4. Confirm the controller is in pass-through / HBA mode rather than presenting a RAID volume: esxcli storage core adapter list
    5. For vSAN ESA, confirm the devices are all-flash NVMe and on the ESA compatibility list, and that the host has the network bandwidth ESA expects (25 GbE): esxcli network nic list

    Expected: Every intended device on every host is visible, unclaimed, and eligible for the chosen vSAN configuration.

  • TP-014 Active Directory Active Directory is reachable and the bind account authenticates E4 · Story 4.3
    1. Confirm each planned domain controller answers on the LDAPS port from the management network: Test-NetConnection <dc-fqdn> -Port 636 (and 389 if plain LDAP is in the design).
    2. Bind with the service account and read the planned base DN — this proves the credential and the base DN in one go. From Linux: ldapsearch -x -H ldaps://<dc-fqdn>:636 -D "<bind-dn>" -W -b "<base-dn>" "(objectClass=user)" dn
    3. From Windows without the AD tools, use ldp.exe (Connection → Connect on 636, then Bind with the service account, then View → Tree on the base DN). A successful bind and a populated tree is the pass.
    4. Check the bind account will not expire underneath the fleet: Get-ADUser <svc-account> -Properties PasswordNeverExpires,PasswordExpired,PasswordLastSet,Enabled | Select Name,Enabled,PasswordNeverExpires,PasswordExpired,PasswordLastSet
    5. Confirm the planned groups exist: Get-ADGroup -Filter "Name -like '<vcf-group-prefix>*'" | Select Name. Then confirm each holds the expected members: Get-ADGroupMember <group-name> | Select -Expand SamAccountName.
    6. Record the bind credential in the secret store, and record WHO owns it.

    Expected: The bind account authenticates against every planned domain controller and can read the intended base DN; the planned groups exist.

    The bind account is a shared dependency across vCenter, NSX, VCF Operations and the Identity Broker — a rotation breaks every consumer at once. Record who owns it.

  • TP-015 Firewall Required firewall flows are open E4 · Story 4.3
    1. Build a host/port list per zone from 07-firewall-ports.md, then test it from a machine ON the source subnet — a rule review is not a test.
    2. Test each flow directly, one command per zone: Test-NetConnection <dc> -Port 636, Test-NetConnection <dns> -Port 53, Test-NetConnection <ca> -Port 443, Test-NetConnection <depot> -Port 443 — confirm TcpTestSucceeded: True for each.
    3. The Cloud Proxy path to VCF Operations specifically — three ports, all needed: Test-NetConnection <vcf-ops-fqdn> -Port 443, -Port 4505, -Port 4506.
    4. From an ESX host where PowerShell is not available: nc -z <host> <port> or esxcli network firewall ruleset list
    5. Confirm UDP flows separately (DNS 53, NTP 123, syslog 514) — Test-NetConnection is TCP only, so use the protocol-level checks from TP-006 and TP-007 instead of assuming.

    Expected: Each zone is proven open by a successful connection from a host on the source subnet.

  • TP-016 Depot Software depot is reachable, or the offline depot is staged and serving Critical E4 · Story 4.3
    1. Connected site — confirm the depot answers from the management network: curl -I https://depot.broadcom.com (add -x http://<proxy>:<port> if a proxy is in use).
    2. Air-gapped site — confirm the offline depot is serving and its certificate is trusted: curl -I https://<offline-depot-fqdn>/ (drop -k; if it only works WITH -k, the certificate is not trusted and the fleet will fail where curl succeeded).
    3. Now confirm the BINARIES are there, not just that the endpoint answers. In VCF Operations: Fleet Management → Lifecycle → Binary Management (or Software Depot) — check the exact target BOM version is listed and shows as downloaded rather than available-to-download.
    4. Cross-check the depot connection itself: VCF Operations → Fleet Management → Lifecycle → Settings → Depot — confirm the URL and credentials are set and the connection status is green.
    5. If a proxy is required, confirm it is stored on the services runtime with tools/Get-VCFProxyConfig.ps1 in this repo.

    Expected: The depot answers from the management network AND the complete target BOM is available from it.

    Reachability alone is a false pass: an empty or partially synced offline depot answers perfectly and then fails at the first lifecycle operation.

  • TP-017 Jump host / BMC Build access: jump host and out-of-band consoles E4 · Story 4.4
    1. From the jump/bastion host, reach each planned appliance address: @("<esxi-1>","<esxi-2>","<vcenter>","<sddc-manager>") | ForEach-Object { Test-NetConnection $_ -Port 443 }.
    2. Open the out-of-band console (iDRAC / iLO / BMC) of EVERY host in a browser and confirm console redirection and virtual media both work — not just that the login page loads.
    3. Check the BMCs answer across the board: @("<bmc-1>","<bmc-2>") | ForEach-Object { Test-NetConnection $_ -Port 443 }.
    4. Confirm the build team has working credentials for both the jump host and the BMCs, and that they are in the secret store.

    Expected: The build team reaches the management network and can open a console on every host.

    Out-of-band configuration is not part of VCF. If the site has no BMC cards, mark NA and record how hosts will be recovered instead.

TP-1 — Management bring-up E5 0/11

Gate: Bring-up completed and the management fleet is healthy.

  • TP-101 ESX Hosts are imaged to the BOM build and correctly networked E5 · Story 5.1
    1. Confirm the build matches the BOM exactly, not just the major version — run this on each host: esxcli system version get
    2. Or across all of them at once from PowerCLI: Get-VMHost | Select Name,Version,Build | Sort Name
    3. Confirm the management VMkernel has the planned IP, mask and gateway: esxcli network ip interface ipv4 get and esxcli network ip route ipv4 list
    4. Confirm the management port group VLAN: esxcli network vswitch standard portgroup list
    5. Confirm DNS and the search domain, then prove resolution works both ways from the host itself: esxcli network ip dns server list then nslookup <this-host-fqdn> and nslookup <its-ip>
    6. Confirm NTP is configured, running, set to start with the host, and actually synced: esxcli system ntp get then esxcli system ntp stats get
    7. Confirm the root password is the planned one — log in with it once — and that it is recorded in the secret store.

    Expected: Every host runs the BOM build, is reachable on its planned management address, resolves DNS both ways, and is time-synced.

  • TP-102 VCF Installer VCF Installer is staged on the VM-Management VLAN under the planned SDDC Manager identity Critical E5 · Story 5.2
    1. Confirm the Installer carries the IP and FQDN planned for SDDC Manager — it becomes SDDC Manager at bring-up, so this is not a throwaway address: Resolve-DnsName <planned-sddc-manager-fqdn> must return the Installer's IP.
    2. Check the port group VLAN on the host running it: esxcli network vswitch standard portgroup list — a fresh host's default VM Network is untagged (VLAN 0). If VM Management is a tagged VLAN, the appliance has NO connectivity until the VLAN ID is set.
    3. Set it if needed: esxcli network vswitch standard portgroup set -p "VM Network" -v <vlan-id>
    4. Open the Installer UI at https://<planned-sddc-manager-fqdn> and confirm it loads over the NAME, not just the IP.
    5. From the Installer, confirm it reaches every ESX host — ping <esxi-mgmt-ip> for each, or run the Installer's own host-validation step.
    6. If the Installer runs OUTSIDE the management domain the wizard deploys a SEPARATE new SDDC Manager appliance and asks for its FQDN. Confirm that extra FQDN and IP are planned and in DNS: Resolve-DnsName <separate-sddc-manager-fqdn>

    Expected: The Installer is reachable on the planned SDDC Manager FQDN, sits on the VM-Management VLAN, and can reach every management host.

  • TP-103 VCF Installer Bring-up completes and the core components are healthy Critical E5 · Story 5.3
    1. On the Installer Review page, DOWNLOAD THE JSON SPEC before starting. It is the repeatable record of what was actually deployed and can be edited and re-uploaded for a repeat run.
    2. Note any soft-stop warning about resource headroom below 20% and record the decision taken.
    3. Run bring-up and watch its milestones: vCenter → SDDC Manager → vSphere cluster → NSX → VCF Management Platform → operations appliance → VCF Management Services. Budget four to six hours.
    4. After completion, confirm vCenter: Connect-VIServer <vcenter-fqdn> then Get-VMHost | Select Name,ConnectionState, and check vCenter → Monitor → Issues and Alarms is clear.
    5. Confirm the NSX Manager cluster: NSX Manager → System → Appliances — three nodes Stable, cluster green, and the VIP answering: Test-NetConnection <nsx-vip-fqdn> -Port 443
    6. Confirm VCF Operations is up at https://<vcf-ops-fqdn>/ (TP-107 checks its cluster in detail).
    7. Confirm SDDC Manager lists the domain: SDDC Manager → Inventory → Workload Domains.

    Expected: Bring-up completes without error; vCenter, SDDC Manager, NSX (3 nodes plus VIP) and VCF Operations are all reachable and healthy.

  • TP-104 vSAN vSAN datastore is online and health checks are clean Critical Automatable E5 · Story 5.3
    1. vSphere Client → Cluster → Monitor → vSAN → Skyline Health → click Retest. Retesting matters: the default view can be a cached run from before the cluster settled.
    2. Work through every red or amber check and either resolve it or record why it is accepted.
    3. Same page → Proactive Tests → run VM Creation Test.
    4. Same page → Proactive Tests → run Network Performance Test.
    5. From the CLI if you prefer: esxcli vsan health cluster list for the check list, and esxcli vsan cluster get to confirm the host is in the cluster with the expected role.
    6. Confirm capacity against sizing: Cluster → Monitor → vSAN → Capacity, or Get-Cluster <name> | Get-Datastore | Select Name,CapacityGB,FreeSpaceGB

    Expected: No failed health checks; both proactive tests pass; datastore capacity matches the sizing output.

  • TP-106 VCF Installer Auto-generated component passwords are captured Critical E5 · Story 5.3
    1. The Installer auto-generates every component password. Retrieve them from the Installer UI → Review Passwords, during or immediately after the deploy.
    2. Store every credential in the engagement secret store.
    3. Cross-check against what SDDC Manager holds, so nothing is missed: .\tools\Get-VCFCredentials.ps1 -SDDCManager <fqdn> -Credential (Get-Credential) — read-only, masked by default.
    4. Confirm each captured credential actually works by logging in with it once. A password captured with a transcription error looks identical to a correct one.

    Expected: Every auto-generated credential is captured, stored in the secret store, and verified by a successful login.

    VCF Operations password management is rotate-and-expire only — there is no reveal API. If these are not captured now, they are not recoverable later.

  • TP-107 VCF Operations VCF Operations cluster is online and consistent Automatable E5 · Story 5.4
    1. Open the admin interface at https://<vcf-ops-fqdn>/admin and log in with the admin account (this is a different UI from the main https://<vcf-ops-fqdn>/).
    2. Go to System Status in the left nav. Under Cluster Status, confirm the cluster reports Online.
    3. On the same page, confirm every node shows state Running and status Online, and that the Cluster Role column matches the design.
    4. Same page — confirm all nodes report the same version and build. A version split is a silent source of odd behaviour.
    5. Same page — confirm each node reports a non-zero Objects and Metrics count.
    6. Same page — confirm SSH Status is off (no green) on every node, unless SSH has been explicitly approved and documented.
    7. If HA or Continuous Availability is in scope, confirm the chosen mode shows Enabled on the same page. They are mutually exclusive — only one should be on.
    8. If the cluster is not yet online, check Software Update in the left nav first: a fresh cluster runs a post-deploy software step and stays offline until it finishes.

    Expected: Cluster online; all nodes running, online, same build, reporting non-zero objects and metrics; SSH off unless approved; the chosen availability mode enabled.

    A new cluster can take a while to come online while its post-deploy software step finishes. Check the update status before treating it as a failure.

  • TP-108 VCF Operations VCF Operations data collection is running Automatable E5 · Story 5.4
    1. In the admin UI (https://<vcf-ops-fqdn>/admin) → System Status — confirm every adapter instance shows Data Receiving.
    2. In the main UI (https://<vcf-ops-fqdn>/) → Administration → Management → Collector Groups — confirm the expected groups exist and each holds the right members.
    3. Main UI → Inventory — confirm every workload domain is listed and is collecting rather than sitting at zero objects.

    Expected: All adapter instances are receiving data, collector groups are correctly populated, and every domain appears in the inventory.

  • TP-109 License Server License Server resolves outside the services-runtime range Critical Automatable E5 · Story 5.4
    1. Confirm the License Server appliance exists and is healthy: VCF Operations → Fleet Management → Lifecycle — it is deployed automatically by the Installer, so this is a verify, not a deploy.
    2. Resolve its FQDN and check the address: Resolve-DnsName <license-server-fqdn>
    3. Confirm that address is IPv4 and falls OUTSIDE the VCF services-runtime range — compare against the services-runtime CIDR from TP-002. Inside the range is a misconfiguration that bites later, not at deploy time.
    4. Note the evaluation start date so the 90-day window is a diarised deadline: VCF Operations → Manage → Licensing → Registration.

    Expected: The License Server is healthy and its FQDN resolves to an IPv4 address outside the services-runtime range.

    This is the bring-up License Server. It is a different appliance from License Hub (TP-411) and the two coexist.

  • TP-110 Cloud Proxy Cloud Proxy is collecting Automatable E5 · Story 5.4
    1. Confirm the Cloud Proxy VM exists and sits on the VM-Management network: Get-VM | Where-Object Name -match "cloud-proxy|cloudproxy" | Select Name,PowerState,@{n="Network";e={($_ | Get-NetworkAdapter).NetworkName}}
    2. From the Cloud Proxy, confirm all three ports to VCF Operations: Test-NetConnection <vcf-ops-fqdn> -Port 443, -Port 4505, -Port 4506.
    3. Confirm it is registered and collecting: VCF Operations → Administration → Management → Collector Groups, or Data Sources → Cloud Proxies — status should be connected with a recent heartbeat, not merely present.

    Expected: The Cloud Proxy is on the VM-Management network, connected, and actively collecting.

  • TP-111 VCF Operations Fleet lifecycle sees the depot, the components and the instances Automatable E5 · Story 5.4
    1. VCF Operations → Fleet Management → Lifecycle — confirm every deployed component is listed.
    2. Same page — confirm the depot is configured and binaries are listed against those components.
    3. Same page — confirm every VCF instance shows as connected.
    4. Fleet Management → Fleet Management → Identity & Access — confirm the identity broker is present (it deploys at bring-up with the management services; configuring it for fleet SSO is TP-422, later).
    5. Confirm the VCF Management Services appliances are up and healthy — services runtime, identity broker, fleet and SDDC lifecycle, software depot, telemetry.

    Expected: All components, the depot with its binaries, and every VCF instance are visible and connected in fleet lifecycle.

  • TP-112 VCF Operations A VCF Operations support bundle can be generated and read E5 · Story 5.4
    1. Admin UI (https://<vcf-ops-fqdn>/admin) → Support → generate a light support bundle for all nodes, then download it.
    2. Open the archive and confirm it holds readable content for EVERY node, not just the primary.
    3. Repeat for a full support bundle from the same page.
    4. For the platform side, run the health check from SDDC Manager over SSH: ssh vcf@<sddc-manager> then cd /opt/vmware/sddc-support and ./sos --health-check

    Expected: Both bundle types generate successfully and contain readable per-node content.

    Slow. Start it in parallel with other work rather than blocking on it — but do prove it now, because the first time you need a bundle is the worst time to discover it fails.

TP-2 — Management configuration E6 0/15

Gate: North-south connectivity, availability, and backup are proven.

  • TP-201 NSX Edge cluster, Tier-0 and BGP are up with routes exchanged Critical Automatable E6 · Story 6.1
    1. NSX Manager → System → Fabric → Nodes → Host Transport Nodes — every host shows Up / Success. Then Edge Transport Nodes and Edge Clusters — edges deployed and the cluster formed.
    2. NSX Manager → Networking → Tier-0 Gateways → your T0 → confirm status is Up and the interfaces sit on the planned uplink VLANs and IPs.
    3. SSH to an edge node (ssh admin@<edge-node>) and check the sessions: get bgp neighbor summary — every neighbour must show Established, with a non-zero uptime.
    4. Check BFD if the design uses it: get bfd-sessions
    5. Now check BOTH directions, which is where the real failure hides. Received: get route bgp. Advertised: get bgp neighbor <peer-ip> advertised-routes
    6. An Established session that exchanges no prefixes is the classic false pass — confirm actual routes, not just the neighbour state.
    7. Confirm the full table the T0 will forward on: get route
    8. From a VM on an overlay segment, reach past the ToRs: ping <upstream-ip> and tracert <upstream-ip> — the path should leave via the T0 uplink.

    Expected: Edge cluster and Tier-0 up; every BGP neighbour established; expected prefixes exchanged in both directions; north-south traffic reaches beyond the ToRs.

    An established session with no prefixes exchanged is a very common false pass. Always check the route tables, not the neighbour state.

  • TP-204 NSX Network segments match the plan Automatable E6 · Story 6.1
    1. NSX Manager → Networking → Segments — compare each segment's name, transport zone, VLAN or overlay type, gateway address and subnet against the network plan.
    2. Confirm no leftover test segments remain.
    3. To diff a long list against the plan rather than reading the UI, pull them from the API: curl -k -u admin:<pw> https://<nsx-manager>/policy/api/v1/infra/segments | ConvertFrom-Json | Select -Expand results | Select display_name,type,vlan_ids,@{n="gw";e={$_.subnets.gateway_address}}

    Expected: Every segment matches the plan, and there are no unplanned leftovers.

  • TP-205 NSX A VM on a segment reaches its gateway and the outside world E6 · Story 6.1
    1. Attach a test VM to each segment and give it an address from that segment: Get-VM TestVM-A | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName "<segment>" -Confirm:$false
    2. From the VM, confirm the gateway answers: ping <segment-gateway>
    3. Confirm the default route is the one you expect: route print (Windows) or ip route (Linux).
    4. Reach outbound past the ToRs: ping <upstream-ip> then tracert <upstream-ip> — an internet address, or an address beyond the rack on an isolated site.
    5. From a machine EXTERNAL to the segment, reach the test VM inbound. Outbound-only is a half-pass and is the more common failure.

    Expected: Gateway, outbound and inbound all succeed on every configured segment.

  • TP-206 ToR fabric Uplinks are clean and the expected VLANs egress the ToRs Automatable E6 · Story 6.1
    1. On the ToRs, check the uplinks for errors, drops and CRCs. Cisco NX-OS: show interface counters errors and show interface <intf> (look at input/output errors and CRC). Arista EOS: show interfaces counters errors.
    2. Clear the counters, wait a few minutes under load, and re-check — a non-zero count from months ago is not the same as errors happening now: clear counters
    3. Confirm the uplinks are up at the expected speed and are not flapping: show interface status and show logging | include LINK
    4. Confirm the VLANs that should egress the ToRs do, and those that should not, do not: show interface trunk on the uplinks.

    Expected: No interface errors on the uplinks, and VLAN egress matches the design in both directions.

  • TP-207 ESX East-west throughput between VMs on the same host E6 · Story 6.1
    1. Put both test VMs on the SAME host: Move-VM -VM TestVM-B -Destination (Get-VM TestVM-A).VMHost
    2. Install iperf3 on both (any recent Linux or Windows build is fine — the point is a consistent tool at both ends).
    3. On the receiver: iperf3 -s
    4. On the sender, with the receiver's REAL address — 4 streams over 30 seconds: iperf3 -c <receiver-ip> -P 4 -t 30
    5. Repeat in reverse to catch a one-way problem: iperf3 -c <receiver-ip> -P 4 -t 30 -R
    6. Now repeat the whole thing with both VMs on an OVERLAY-backed segment instead of the VDS port group.
    7. Record the measured numbers (VDS and overlay, both directions) in the actual-result box. A number, not a tick.

    Expected: Same-host throughput reaches the level agreed for this platform on both VDS and overlay networking, with no packet loss.

    Set the target with the customer before testing. A same-host figure is bounded by CPU and vNIC, not the physical fabric, so it should comfortably exceed the cross-host figure.

  • TP-208 ESX East-west throughput between VMs on different hosts E6 · Story 6.1
    1. Split the pair across hosts: Move-VM -VM TestVM-B -Destination <other-host>
    2. Repeat the VDS-backed test: iperf3 -s on the receiver, iperf3 -c <receiver-ip> -P 4 -t 30 on the sender, then -R for the reverse direction.
    3. Repeat on an OVERLAY-backed segment. This is the path that actually exercises the host overlay (TEP-to-TEP) and its MTU.
    4. If the overlay figure is far below the VDS figure, suspect the overlay MTU rather than the fabric — go back and re-run the TP-001 check against the TEP VMkernel: vmkping -I vmk10 -d -s 8972 <peer-tep-ip> (esxcli network ip interface ipv4 get to find the TEP vmk).
    5. Record all four measured numbers in the actual-result box.

    Expected: Cross-host throughput reaches the agreed level on both VDS and overlay, with no packet loss and no fragmentation.

    A large gap between the VDS and overlay figures usually means the host overlay MTU is not what TP-001 thought it was.

  • TP-209 vCenter VMs deploy and anti-affinity rules apply E6 · Story 6.1
    1. Deploy two test VMs. From an OVF: vSphere Client → right-click the cluster → Deploy OVF Template → name them TestVM-A and TestVM-B, storage <datastore>, network <mgmt-portgroup>.
    2. Or from a template with PowerCLI: New-VM -Name TestVM-A -Template <template> -ResourcePool <cluster> -Datastore <datastore>, then the same for TestVM-B.
    3. Power both on and confirm they boot and take an address: Get-VM TestVM-A,TestVM-B | Select Name,PowerState,VMHost,@{n="IP";e={$_.Guest.IPAddress -join ","}}
    4. Create the anti-affinity rule: New-DrsRule -Cluster <cluster> -Name "TestVMs-AntiAffinity" -KeepTogether $false -VM (Get-VM TestVM-A,TestVM-B)
    5. Confirm it exists and is enabled: Get-DrsRule -Cluster <cluster> | Select Name,Enabled,KeepTogether — or Cluster → Configure → VM/Host Rules.
    6. Confirm DRS actually separated them: Get-VM TestVM-A,TestVM-B | Select Name,VMHost — two different hosts.

    Expected: Both VMs deploy successfully and the anti-affinity rule is created and applied.

    These two VMs are reused by TP-207, TP-208, TP-210 and TP-211. Build them once, early.

  • TP-210 vCenter vMotion works between every pair of hosts E6 · Story 6.1
    1. Start a continuous ping to the test VM from elsewhere so you can see any drop during migration: ping -t <testvm-ip> (Windows) or ping <testvm-ip> (Linux).
    2. Migrate it, compute resource only: Move-VM -VM TestVM-A -Destination <target-host> — or vSphere Client → right-click the VM → MigrateChange compute resource only.
    3. Confirm the VM stayed up: the ping should lose at most one packet.
    4. Now walk it round every host so each one both sends and receives — a single migration proves one path, not the cluster: Get-Cluster <name> | Get-VMHost | ForEach-Object { Move-VM -VM TestVM-A -Destination $_ -Confirm:$false; "moved to $($_.Name)" }
    5. Confirm the anti-affinity rule blocks a co-locating move: try Move-VM -VM TestVM-B -Destination (Get-VM TestVM-A).VMHost and confirm DRS refuses it or flags the violation.

    Expected: Every host both sends and receives a live migration with no interruption, and the anti-affinity rule blocks a co-locating move.

  • TP-211 vCenter vSphere HA restarts workloads after a host failure Critical E6 · Story 6.1
    1. Confirm HA is on and admission control matches the design: Get-Cluster <name> | Select Name,HAEnabled,HAAdmissionControlEnabled,HAFailoverLevel,DrsEnabled,DrsAutomationLevel
    2. Note which host currently runs the test VM: Get-VM TestVM-A | Select Name,VMHost
    3. Start a continuous ping to the VM so you can time the outage: ping -t <testvm-ip>
    4. HARD-power that host off from its out-of-band console (iDRAC / iLO / BMC) — power off, NOT a graceful shutdown, which does not exercise HA at all.
    5. Watch the VM restart elsewhere: while ($true) { Get-VM TestVM-A | Select Name,VMHost,PowerState; Start-Sleep 15 } (Ctrl+C to stop once it reports back on the new host).
    6. Record the outage duration from the ping output — that is your measured recovery time, and it goes in the actual-result box.
    7. Power the host back on and confirm it rejoins cleanly with no lingering alarms: Get-VMHost <host> | Select Name,ConnectionState,PowerState
    8. For vSAN, confirm resync finished before moving on: vSphere Client → Cluster → Monitor → vSAN → Resyncing Objects should be empty.
    9. Repeat for any additional cluster in scope.

    Expected: The VM restarts on a surviving host within the agreed recovery time, and the failed host rejoins cleanly.

  • TP-212 SDDC Manager SDDC Manager backup completes to the SFTP target Critical Automatable E6 · Story 6.4
    1. Check what the platform actually stored — target, username, schedule, retention — with tools/Get-VCFBackupConfig.ps1 from this repo. That reads the Fleet LCM API, so it shows the real configuration rather than what the wizard appeared to accept.
    2. Or in the UI: VCF Operations → Fleet Management → Lifecycle → Backup Configuration.
    3. Confirm the SFTP target answers from the management network: Test-NetConnection <sftp-host> -Port 22
    4. Trigger a backup and watch it to completion (UI: Backup Configuration → Backup Now, then follow the task).
    5. Now go and look on the TARGET — a task that reports success and an archive that exists are different claims: sftp <user>@<sftp-host> then ls -l <backup-path>. Confirm a new archive with a plausible size and a current timestamp.
    6. Confirm schedule and retention match what was agreed.
    7. Confirm the backup encryption passphrase is in the secret store. Without it the archive is unusable, and nothing will tell you that until a restore.

    Expected: A backup completes, the archive is present on the SFTP target, the schedule is set, and the passphrase is stored.

  • TP-213 vCenter Every vCenter file-based backup is scheduled and completes Critical Automatable E6 · Story 6.4
    1. List every vCenter in the fleet so none is missed: Get-VIServer / the inventory in VCF Operations → Fleet Management → Lifecycle.
    2. For EACH one, open its own VAMI at https://<vcenter-fqdn>:5480 and log in as root. This is per-vCenter and manual — VCF does not configure it for you.
    3. Go to Backup in the left nav and confirm a schedule exists: target URL (e.g. sftp://<host>/<path>), credentials, day/time, retention, and which parts are included.
    4. If it is missing, create it: Backup → Configure → enter the target, credentials, schedule, retention, and an encryption password.
    5. Run one now: Backup → Backup Now → confirm it completes and note the archive name.
    6. Verify on the target that each vCenter wrote its own folder: sftp <user>@<sftp-host> then ls -l <backup-path> — expect one subfolder per vCenter.
    7. Record every encryption password in the secret store, per vCenter.

    Expected: Every vCenter has a working, scheduled file-based backup with its archive verified on the target.

    This is the single most commonly missed post-deployment task: SDDC Manager backup is configured centrally and looks like it covers everything, but each vCenter must be set up individually in its own VAMI.

  • TP-214 Backup target Backups are restorable, not merely present E6 · Story 6.4
    1. Pull one archive down to a separate machine: sftp <user>@<sftp-host> then get -r <backup-path>/<latest-folder>
    2. Confirm it is complete and not truncated — compare the byte count against what the source reported, and confirm every expected file is present.
    3. Confirm the passphrase from the secret store actually opens it. This is the step that catches a passphrase recorded with a typo, which is otherwise invisible until the day it matters.
    4. Confirm the restore procedure is written down: where the archive lives, where the passphrase lives, who can reach the target, and the restore steps per component.

    Expected: An archive can be retrieved and opened with the stored passphrase, and the restore procedure is written down.

    An unverified backup is a hypothesis. This is the cheapest possible version of testing it — do at least this much.

  • TP-215 Depot Fleet lifecycle depot is connected and pending updates are understood Automatable E6 · Story 6.4
    1. VCF Operations → Fleet Management → Lifecycle → Settings → Depot — confirm the fleet-wide depot is connected. This is separate from the depot SDDC Manager already got at bring-up, not a re-do of it.
    2. Confirm the proxy if the site needs one: .\tools\Get-VCFProxyConfig.ps1 from this repo reads what is actually stored on the services runtime, rather than what the wizard appeared to accept.
    3. Fleet Management → Lifecycle → Binary Management — confirm bundles for the current BOM are present and downloaded.
    4. Review available updates: Lifecycle → <SDDC Manager> → Updates. Record the decision for each — applied now, scheduled, or deferred with a reason. An unreviewed update list is not a pass.

    Expected: The fleet lifecycle depot is connected and every available update has a recorded decision.

  • TP-216 vSAN Storage firmware and driver health is clean Automatable E6 · Story 6.4
    1. vSphere Client → Cluster → Monitor → vSAN → Skyline Health → the Hardware compatibility section — check the controller firmware and driver items specifically, not just the overall status.
    2. Confirm the hosts are set up to receive firmware and driver updates through the intended channel: Cluster → Updates → Image (or the HCL database refresh setting if the site is offline).
    3. Confirm the installed driver/firmware from the host side: esxcli storage core adapter list then vmkload_mod -s <driver> | grep -i version
    4. Resolve or document every warning — an accepted firmware mismatch needs a written reason, not a silent pass.

    Expected: No firmware or driver health failures, and the update channel is configured.

  • TP-217 vCenter Early identity binding behaves as designed Automatable E6 · Story 6.3
    1. If vCenter SSO was bound directly to AD/LDAP for early access: vCenter → Administration → Single Sign On → Configuration → Identity Provider — confirm the source is configured and its connection test passes.
    2. Log in as a directory user and confirm the role that lands: Administration → Access Control → Global Permissions should show the expected group-to-role mapping.
    3. Confirm the admin, operator and viewer group mappings match the design — test one user per role, do not read the configuration.
    4. Confirm privileges on the non-vSphere components: NSX Manager → System → Users and Roles, and VCF Operations → Administration → Access → Access Control.

    Expected: Directory logins land on the intended roles across vCenter, NSX and VCF Operations.

    Fleet-wide SSO via the Identity Broker (TP-430) is the recommended path and is done Day-2. Mark this NA if identity was deliberately deferred.

TP-4 — Day-2 fleet E8 0/19

Gate: Every deferred and added component is healthy, certified, licensed and integrated.

  • TP-401 NSX Fleet component network placement is built and reachable E8 · Story 8.1
    1. Confirm the chosen placement exists — shared management, dedicated management, an NSX overlay segment, or an NSX VLAN segment. For an NSX-backed placement: NSX Manager → Networking → Segments.
    2. If a non-shared network was built, confirm it is routed and reachable from the management network: Test-NetConnection <an-ip-on-that-network> -Port 443, and tracert to confirm the path is what you expect.
    3. Confirm every Day-2 appliance FQDN (VCF Automation, Log Management VIP, VCF Operations for Networks) resolves forward AND reverse onto that network — same method as TP-006: Resolve-DnsName <fqdn> -Type A, then Resolve-DnsName <ip> -Type PTR.

    Expected: The chosen placement is built and routed, and all Day-2 FQDNs resolve both ways onto it.

    An overlay-segment placement needs an edge cluster and a Tier-0. Under Distributed connectivity there is none, so either build one for the fleet segment or pick a VLAN-backed placement.

  • TP-402 VCF Automation Automation appliances: identity, time and cluster state Automatable E8 · Story 8.2
    1. SSH to each appliance with the expected credential: ssh root@<vcfa-appliance>
    2. Confirm time zone and NTP: vracli ntp status — check the configured server and that it is synced, not just set.
    3. Confirm the hostname is the correct FQDN: vracli status and read hostname under the host Nodes section.
    4. Confirm that name resolves both ways from a client: Resolve-DnsName <vcfa-fqdn> then Resolve-DnsName <its-ip>
    5. Confirm the cluster is ready: vracli status and look for a status entry of type Ready with "status": "True".

    Expected: Every appliance authenticates, is time-synced, carries the correct FQDN resolving both ways, and the cluster reports ready.

  • TP-403 VCF Automation Automation is reachable on its cluster VIP Critical E8 · Story 8.2
    1. Open the provider portal through the VIP: https://<vcfa-fqdn>/provider — log in as a system administrator. The built-in load balancer is configured automatically for both the single-node and the HA model; no external load balancer is required to reach this.
    2. Confirm the VIP is what you connected to, not one node: Resolve-DnsName <vcfa-fqdn> should return the cluster VIP.
    3. On an HA cluster, prove the VIP survives a node loss: Stop-VM -VM <one-vcfa-node> -Confirm:$false, then reload the portal. It must keep serving.
    4. Power the node back on and confirm it rejoins: vracli status shows all three nodes ready again.
    5. Confirm the services-runtime cluster CIDR is set and overlaps nothing — cross-check against the subnet list from TP-002.

    Expected: The portal is reachable and serving through the cluster VIP, survives the loss of one node on HA, and the services-runtime CIDR is unique.

  • TP-404 VCF Automation Provider plane matches the design E8 · Story 8.2
    1. Provider portal → Infrastructure → Networking → Provider Gateways — confirm the gateway is discovered, its type shows as the expected T0/VRF, and its status is Normal.
    2. Click Associated IP Spaces on that gateway — confirm each IP space reports Normal.
    3. Provider portal → Infrastructure → Regions — confirm each region reports Normal and shows the expected supervisor and NSX Manager.
    4. Provider portal → Infrastructure → Networking → Edge Clusters — confirm health is Healthy where applicable.
    5. Administration → Connections → Virtual Centers — confirm each vCenter is Enabled, Connected, and Licensed (not just connected — an unlicensed vCenter connects fine and fails later).
    6. Administration → Connections → NSX Managers — confirm status Normal. These are auto-discovered through VCF Operations, so an absent one means the discovery path is broken rather than that you forgot to add it.
    7. Administration → VCF Instances — confirm the VCF name, hostname and version are as expected.

    Expected: Provider gateway, IP spaces, regions and connections all report normal, and the vCenter connections report licensed.

  • TP-405 VCF Automation Content libraries and their items are ready E8 · Story 8.2
    1. Provider portal → Infrastructure → Content Libraries — confirm the expected libraries are present, with the correct local or subscribed type, and status Ready.
    2. Content Library Items — confirm the expected templates/OVAs are discovered and each shows Ready.
    3. On a subscribed library, trigger a sync: the three-dot menu → Sync Library — and confirm it completes rather than erroring on the subscription URL or its certificate.

    Expected: Libraries and their items are present and ready, and a subscribed library syncs successfully.

  • TP-406 VCF Automation Organizations, projects, quotas and membership are configured E8 · Story 8.2
    1. Provider portal → Infrastructure → Organizations — confirm the expected organizations exist with their allocated regions and resources.
    2. Organization portal (https://<vcfa-fqdn>/org/<org-name>) → Manage & Govern → Projects — confirm projects exist with the intended type, quotas and resource limits.
    3. Administer → Access Control → Users and Groups — confirm they come from the identity provider rather than being local accounts, and land on the intended roles.
    4. Administer → Access Control → Roles — confirm the role set matches the design, including any custom roles and their rights.
    5. Manage & Govern → Policies → Definitions — confirm quota, lease and approval policies are configured as designed.

    Expected: Organizations, projects, quotas and role assignments all match the design, sourced from the identity provider.

  • TP-407 VCF Automation End-to-end provisioning: request, deploy and decommission Critical E8 · Story 8.2
    1. Note the current IP pool usage BEFORE you start, so you can prove the cleanup later: Manage & Govern → Networking → IP Management → IP Address Blocks — record the allocated count.
    2. As an end user: Consume → Catalog → pick an item → Request → fill in the deployment name and inputs → Submit.
    3. Watch it through: Consume → Deployments → the deployment → History. Check for warnings, not just the final success.
    4. Confirm the workload is actually usable, not merely created: ping it, confirm it has the right address on the right network, and confirm CPU/RAM/disk match what the blueprint specified.
    5. Perform a day-2 action from Consume → Deployments → Actions — a power off and power on is enough.
    6. Now DELETE the deployment: Actions → Delete.
    7. Confirm the cleanup is complete: the VMs are gone from vCenter (Get-VM <deployment-name>*), any created networks are gone, and — the one people miss — the IP allocation count is back to where you noted it.

    Expected: A catalog request deploys a working workload, day-2 actions succeed, and deletion removes every resource it created with no orphans left behind.

    Decommission is the half people skip. A blueprint that deploys but leaks IP allocations on delete will exhaust the pool weeks after handover.

  • TP-415 Log Management Log Management appliances are deployed, networked and resolvable Automatable E8 · Story 8.4
    1. Confirm every appliance is powered on and on the right network: Get-VM | Where-Object Name -match "<log-appliance-prefix>" | Select Name,PowerState,@{n="Network";e={($_ | Get-NetworkAdapter).NetworkName}},@{n="HW";e={$_.HardwareVersion}}
    2. Confirm forward and reverse DNS for the cluster VIP AND for every individual node — both matter, because clients hit the VIP but the nodes talk to each other by name. Same method as TP-006, run against each of <vip-fqdn>, <node1-fqdn>, <node2-fqdn>, <node3-fqdn>: Resolve-DnsName <fqdn> -Type A, then Resolve-DnsName <ip> -Type PTR.
    3. Confirm the virtual hardware version suits the appliance size — a large appliance needs a version that supports its vCPU count.
    4. Open https://<log-vip-fqdn> and confirm the integrated load balancer is serving it. Unlike VCF Operations, this product genuinely has a cluster VIP.

    Expected: All nodes are up on the correct network, the VIP and every node resolve both ways, and the VIP serves the interface.

    This product has an integrated load balancer and a genuine cluster VIP. VCF Operations does not — do not confuse the two.

  • TP-416 Log Management Log Management access control and integrations are correct Automatable E8 · Story 8.4
    1. Administration → Authentication → Active Directory — confirm the configuration and click Test Connection, if directory integration is in scope.
    2. Administration → Access Control → Users and Groups — confirm the expected accounts and groups exist.
    3. Check membership of EVERY role group, not just the admin one: Super Admin, View Only Admin, Dashboard User, and User. An over-broad membership here is the finding.
    4. Administration → Time — confirm the NTP servers match the customer's.
    5. Administration → SMTP — confirm server, port, SSL/STARTTLS and sender, then send a test message.
    6. Administration → Archiving — confirm archiving is enabled and the NFS archive location is set and writable.
    7. Administration → SSL — confirm the certificate is the intended one, not the self-signed default (unless a self-signed cert was the agreed outcome).
    8. Administration → Event Forwarding — confirm each destination's protocol, port, SSL, filter, tag, disk cache and worker count match the design.

    Expected: Directory integration, group membership, NTP, SMTP, archiving, SSL and event forwarding all match the design.

  • TP-417 Log Management Every source is actually shipping logs to Log Management Critical Automatable E8 · Story 8.4
    1. Administration → vSphere Integration — confirm each vCenter is configured against the cluster VIP.
    2. Confirm EVERY ESX host points at the VIP, not a sample: Get-VMHost | Select Name,@{n="Syslog";e={($_ | Get-AdvancedSetting -Name Syslog.global.logHost).Value}} | Sort Name
    3. Any host with a blank or wrong Syslog.global.logHost is a silent gap — it will be invisible until the day you need that host's logs.
    4. Now prove delivery, which is a different claim from configuration. In the UI go to Explore Logs, filter by hostname for a specific ESX host, and confirm events from the last few minutes.
    5. Repeat the check for a vCenter source.
    6. Generate a real event to test end to end: restart syslog on a host (esxcli system syslog reload) or log a test message, then find it in Explore Logs.
    7. Create a test alert query under Explore Logs → Alerts, trigger it, and confirm the email or integration notification actually arrives.

    Expected: Every vCenter and every ESX host is shipping logs to the VIP with recent events visible, and a triggered test alert is delivered.

    Configuration and delivery are different things. Check for arriving events per source, because one silently missing host is invisible until the day you need its logs.

  • TP-418 Ops for Networks Operations for Networks is paired, collecting and healthy Automatable E8 · Story 8.4
    1. Confirm the platform and collector VMs are up on the right network: Get-VM | Where-Object Name -match "<vcfon-prefix>" | Select Name,PowerState,@{n="Network";e={($_ | Get-NetworkAdapter).NetworkName}}
    2. Confirm forward and reverse DNS for every appliance address.
    3. Settings → Install and Support — confirm every collector (proxy) VM is listed under Collector VMs and paired with the platform, with no warnings.
    4. Same page — confirm the Overview health indicator is green, and that platform VMs are listed without warnings if running clustered.
    5. Settings → Data Sources — confirm every intended source is connected and collecting, with no warning icons. A source added but failing to collect shows here, not on the health page.
    6. Settings → Licensing — confirm a real licence is applied. The specific check is that you CANNOT switch to assessment mode; if you can, the instance is running on an evaluation licence.
    7. Confirm NTP, SMTP and the SSL certificate under Settings match the design.

    Expected: Platform and collectors are paired and healthy, all data sources collect without warnings, and the instance is licensed rather than in assessment mode.

  • TP-419 Ops for Networks Operations for Networks detects a real change and is usable for analysis E8 · Story 8.4
    1. Make a small, reversible change on a monitored object — a dummy port group is the easiest: New-VirtualPortGroup -VirtualSwitch <vds> -Name "vcfon-test-pg" -VLanId 999
    2. Wait for the next collection cycle, then find it: Search → query for change events in the last hour, or open Events and filter for Configuration Change.
    3. Confirm the configured notification (email or SNMP trap) actually arrived.
    4. Revert the change: Remove-VirtualPortGroup -VirtualPortGroup "vcfon-test-pg" -Confirm:$false
    5. Run a representative search in Search, e.g. vms where cpu count > 1, then pin the result and confirm it appears under Pinboards.
    6. Run Path and Topology between two workloads and confirm the path renders end to end.

    Expected: A real configuration change is detected and notified, and search, pinboards and path analysis all work.

  • TP-420 VCF Operations Fleet certificates are CA-signed with no trust warnings Critical Automatable E8 · Story 8.5
    1. Confirm the replacement was run in STAGGERED batches, letting each finish before starting the next. Each rotation triggers automated retrust across dependent components, and the UI makes you acknowledge exactly that.
    2. Check each endpoint from the command line rather than clicking through browsers — issuer, dates and SANs in one call: openssl s_client -connect <fqdn>:443 -servername <fqdn> < NUL 2>NUL | openssl x509 -noout -issuer -dates -ext subjectAltName
    3. Confirm the issuer is the intended CA on every endpoint — anything still showing a self-signed or VMCA issuer was missed by the batch.
    4. Confirm the SANs carried through, especially any load-balancer VIP FQDN.
    5. Confirm no browser trust warning on each endpoint (this is what the customer will see).
    6. Now confirm inter-component trust SURVIVED the rotation, which browsers cannot tell you: VCF Operations → Fleet Management → Lifecycle should still show every instance connected, and Administration → Connections in VCF Automation should still show vCenter and NSX connected.

    Expected: Every endpoint presents a CA-signed certificate with correct SANs and no trust warning, and all inter-component integrations still report connected.

    Where the load balancer is managed through VCF Operations, drive its certificate from there. Swapping it directly in the product UI breaks NSX trust.

  • TP-421 VCF Operations Certificate auto-renewal is configured E8 · Story 8.5
    1. VCF Operations → Fleet Management → Fleet Management → Certificates — confirm the expected certificate authority is configured.
    2. On the same page, confirm auto-renewal is enabled where intended.
    3. Record the EARLIEST expiry date across the fleet — from the sweep in TP-420 — and confirm it is diarised with the customer.
    4. If the list looks empty, wait before calling it a failure: objects can take six hours or more to appear under certificate management after deployment.

    Expected: The CA is configured, auto-renewal is enabled where intended, and the earliest expiry date is known and diarised.

    Objects can take several hours to appear under certificate management after deployment. Do not read an empty list as a failure straight away.

  • TP-422 Identity Broker Fleet SSO through the identity broker works end to end Automatable E8 · Story 8.5
    1. VCF Operations → Fleet Management → Fleet Management → Identity & Access — confirm every VCF instance (vCenter and NSX) and every VCF management appliance is configured against the intended identity broker. A component missing from this list is not covered by fleet SSO.
    2. Log in as a directory user and confirm single sign-on carries across the fleet components without re-prompting.
    3. Confirm group-to-role mapping: log in as one test user per role (admin, operator, viewer) and confirm each lands on the intended role in each component. Mapping that looks right in configuration and lands wrong in practice is common.
    4. Confirm the break-glass LOCAL account works — log in with it once, right now, while everything is healthy. The moment you need it is the moment you cannot test it.
    5. Confirm the break-glass credential is recorded in the secret store, out of band from the identity provider it exists to work around.

    Expected: Directory users sign on once and reach every fleet component on the intended role, and a working break-glass local account is recorded.

    The identity broker itself is deployed at bring-up with the management services. This is its configuration, not its deployment.

  • TP-423 VCF Operations Negative test: an unprivileged user cannot administer Critical E8 · Story 8.5
    1. Log in as a deliberately NON-administrative user. WRITE DOWN which account you used in the actual-result box — this is the step most often got wrong, and a negative test run as admin passes vacuously and proves nothing.
    2. VCF Operations → Administration → Solutions → Policies — attempt to create or modify a policy. It must be refused.
    3. VCF Operations → Administration → Access → Access Control → User Accounts — attempt to create a user, and then a group. Both must be refused.
    4. Confirm the user cannot see objects outside their intended scope — check the inventory shows only what their role permits.
    5. In vCenter, confirm the same user cannot see the VCF Operations plug-in menu or objects they have no rights to.
    6. Repeat on each component with its own role system: vCenter, NSX, VCF Operations, VCF Automation, and the log and network products if deployed.

    Expected: Every administrative action is refused for the non-administrative user, on every component tested.

    Write down which account was used. A negative test executed with an admin account passes vacuously and proves nothing.

  • TP-424 VCF Operations Licensing is applied across the fleet Automatable E8 · Story 8.5
    1. VCF Operations → Manage → Licensing → Registration — confirm the environment is registered.
    2. Manage → Licensing → License Keys — confirm every key is installed and valid, and that the type, capacity, usage and expiry match the customer's entitlement.
    3. Manage → Licensing → License Groups — confirm the groups exist and hold the right members and member types.
    4. Confirm each vCenter carries its primary licence and any add-ons: Get-VIServer | Select Name then check each in vCenter → Administration → Licensing → Assets.
    5. Confirm NOTHING is still in evaluation. The 90-day window started at bring-up, so check the actual expiry rather than assuming there is time.
    6. Check the other products separately — they license independently: the log platform under Administration → Licensing, and the network platform under Settings → Licensing.

    Expected: All licences are installed and valid, groups are correct, and nothing is left in evaluation.

    The evaluation window starts at bring-up and is 90 days. Confirm the expiry date rather than assuming there is time.

  • TP-425 VCF Operations Operations alerting and self-health are sound E8 · Story 8.5
    1. VCF Operations → Alerts — review what is active. Either there are none, or every one has a written explanation.
    2. Exercise the workflow on one alert: take ownership (control state becomes Assigned, assigned user becomes you), release ownership (returns to Open), then cancel one (status becomes Inactive).
    3. If a ticketing integration is in scope, configure an alert rule with that notification method, trigger a matching alert, and confirm the incident is actually raised in the target system.
    4. Dashboards → Self Health, Self Cluster Statistics, Self Performance Details and Self Services Communications — confirm each reports green, and explain anything that does not.
    5. If those dashboards are missing, enable them: Dashboards → VCF Operations section.
    6. Administration → Management → Log Forwarding — confirm forwarding to the log platform is configured, then confirm events are ARRIVING there, not just that the config exists.
    7. Administration → Management → Outbound Settings — confirm the automated actions plug-in shows active if it is in scope.

    Expected: No unexplained active alerts, the alert workflow and any ticketing integration work, self-health is green, and log forwarding is arriving.

  • TP-426 VCF Operations Operations content is usable by the customer E8 · Story 8.5
    1. Dashboards → Create Dashboard — drag a few widgets on, name it, save.
    2. Select it → Export — confirm a dashboard file downloads.
    3. Import it back and confirm it succeeds.
    4. Select it → Share → drag it to a group. Log in as a member of that group and confirm they can SEE it but not edit or delete it.
    5. Back as admin, Stop Sharing, then confirm the member has lost access.
    6. Save it as a template and confirm it appears for other users to create from.
    7. Administration → Solutions → Policies — confirm the expected policies exist, are active, and are assigned to the right groups. Check the policy library tree structure matches the design.
    8. Administration → Configuration → Maintenance Schedules — confirm the schedules match what the customer agreed, and that they have confirmed them.
    9. Administration → Management → Global Settings → Currency — confirm it is set correctly for the customer, since it cannot be changed once cost data accumulates.

    Expected: Dashboards create, export, import and share correctly; policies are active and correctly assigned; maintenance schedules match the agreement.

TP-5.1 — Workload domain — wld01 E9 0/6

Gate: The domain is healthy, north-south works, and workloads can be placed.

  • TP-501 ToR fabric Workload domain network prep is complete E9 · Story 9.1
    1. Confirm this domain's VLANs are trunked to ITS hosts at the correct MTU — on the ToRs show interface trunk, then prove the MTU from a host: vmkping -I vmk1 -d -s 8972 <peer-vmk-ip> (across both AZs if this domain is stretched).
    2. Confirm the addresses this domain consumes on the management VM-management subnet are reserved in IPAM — a workload domain takes several.
    3. Confirm forward AND reverse DNS for this domain's vCenter, NSX components and hosts — same method as TP-006, run against each of <wld-vcenter>, <wld-nsx-vip>, <wld-esxi-1>: Resolve-DnsName <fqdn> -Type A, then Resolve-DnsName <ip> -Type PTR.

    Expected: This domain’s VLANs, subnets, reservations and DNS records are all in place and resolve both ways.

  • TP-502 ESX Workload domain hosts are imaged and commissioned E9 · Story 9.2
    1. Confirm each host runs the BOM build: esxcli system version get per host, or Get-VMHost | Select Name,Version,Build | Sort Name once they are in an inventory.
    2. Confirm the model, controller and NICs are on the compatibility guide at their installed firmware/driver level.
    3. Confirm the management network, DNS and NTP on each — per AZ if this domain is stretched: esxcli network ip interface ipv4 get, esxcli network ip dns server list, esxcli system ntp stats get
    4. SDDC Manager → Inventory → Hosts — confirm they are commissioned and show as available for this domain.

    Expected: All hosts run the matched build, are correctly networked and time-synced, and are commissioned and available.

  • TP-503 SDDC Manager Workload domain is deployed and healthy Critical Automatable E9 · Story 9.3
    1. SDDC Manager → Inventory → Workload Domains → View Details — confirm this domain appears with its CPU, memory and storage allocation.
    2. Open its vCenter and confirm it is reachable and healthy: Connect-VIServer <wld-vcenter-fqdn> then Get-VMHost | Select Name,ConnectionState,PowerState
    3. Confirm its NSX — shared or dedicated per the design — is healthy: NSX Manager → System → Appliances, all nodes stable and the VIP answering.
    4. Confirm the first cluster is online with every host joined: Workload Domains → the domain → Clusters → the cluster → Hosts.
    5. Run the platform health check and review: ssh vcf@<sddc-manager>, then cd /opt/vmware/sddc-support and ./sos --health-check --domain-name <wld-name>

    Expected: The domain, its vCenter, its NSX and its first cluster are all deployed and healthy.

  • TP-504 Principal storage Workload domain principal storage is healthy Critical Automatable E9 · Story 9.3
    1. vSAN — Cluster → Monitor → vSAN → Skyline HealthRetest, then resolve or document every failure. Run Proactive Tests → VM Creation Test and Network Performance Test.
    2. NFS or FC — confirm the datastore mounts on every host in THIS cluster — list which hosts see it: Get-Cluster <wld-cluster> | Get-VMHost | Get-Datastore -Name <datastore>. The row count must equal the cluster's host count; any host missing from the output can't see the datastore.
    3. FC — confirm full path redundancy: esxcli storage core path list -d <naa.id> | grep -E "Runtime Name|State"
    4. Confirm capacity matches the sizing for this domain: Get-Cluster <wld-cluster> | Get-Datastore | Select Name,CapacityGB,FreeSpaceGB

    Expected: The principal datastore is healthy and writable across every host in the cluster, with capacity matching sizing.

  • TP-505 NSX Workload domain north-south via edge cluster and BGP Critical Automatable E9 · Story 9.4 / 9.6
    1. NSX Manager → System → Fabric → Nodes — confirm THIS domain's hosts are transport nodes and this domain's edge cluster is deployed.
    2. NSX Manager → Networking → Tier-0 Gateways — confirm this domain's T0 is Up on the planned uplinks.
    3. SSH to this domain's edge node and check both directions: get bgp neighbor summary for Established sessions, get route bgp for received prefixes, get bgp neighbor <peer-ip> advertised-routes for what is being sent.
    4. From a VM on this domain's overlay, reach past the ToRs and back: ping <upstream-ip> and tracert <upstream-ip>.

    Expected: This domain’s edge cluster and Tier-0 are up, BGP is exchanging prefixes both ways, and north-south works end to end.

    Connectivity is chosen per workload domain, independently of the management domain. This domain gets its own edge cluster.

  • TP-507 vCenter Workload domain datapath and availability E9 · Story 9.4 / 9.6
    1. Deploy two test VMs into this domain and apply an anti-affinity rule: New-DrsRule -Cluster <wld-cluster> -Name "TestVMs-AntiAffinity" -KeepTogether $false -VM (Get-VM TestVM-A,TestVM-B)
    2. Walk a running VM round every host so each both sends and receives: Get-Cluster <wld-cluster> | Get-VMHost | ForEach-Object { Move-VM -VM TestVM-A -Destination $_ -Confirm:$false; "moved to $($_.Name)" } — with a ping -t running to confirm no interruption.
    3. HARD-power a host off from its out-of-band console, confirm HA restarts its workload on a survivor, then power it back on and confirm it rejoins: Get-VMHost <host> | Select Name,ConnectionState
    4. Measure throughput with iperf3, same host and different hosts, on both VDS-backed and overlay-backed networks: iperf3 -s on the receiver, iperf3 -c <receiver-ip> -P 4 -t 30 then -R on the sender.
    5. Record all the measured throughput figures in the actual-result box.

    Expected: Migrations succeed across every host, HA restarts workloads after a hard failure, and throughput meets the agreed level on both same-host and cross-host paths.

TP-6 — Validation & handover E10 0/7

Gate: The complete environment is verified, evidenced, and signed off.

  • TP-601 SDDC Manager Final health check of the complete environment Critical Automatable E10 · Story 10.1
    1. Re-run the platform health check against the FINAL state — an early clean run does not describe the environment being handed over: ssh vcf@<sddc-manager>, cd /opt/vmware/sddc-support, ./sos --health-check
    2. Re-check storage health on EVERY cluster in EVERY domain: vSphere Client → Cluster → Monitor → vSAN → Skyline HealthRetest, per cluster.
    3. Confirm no version or build inconsistency across the estate: Get-VMHost | Group-Object Version,Build | Select Count,Name — more than one group inside a cluster is a finding.
    4. VCF Operations → Fleet Management → Lifecycle — confirm every component is at its expected version and nothing is mid-upgrade.
    5. Triage every finding: fixed, accepted with a written reason, or scheduled with a named owner. Nothing left unexplained.

    Expected: The final-state health check is clean, or every finding is triaged with an owner and a decision.

  • TP-602 VCF Operations No unexplained active alerts Automatable E10 · Story 10.1
    1. VCF Operations → Alerts — review everything active across the fleet, filtered to Critical and Immediate first.
    2. For each, either resolve it or record why it is expected and accepted. "It was there yesterday too" is not a reason.
    3. Dashboards → Self Health — confirm green, and explain anything that is not.
    4. Check the per-component alert views too, since they do not all roll up: vCenter → Monitor → Issues and Alarms, NSX Manager → System → Alarms, and the log and network products if deployed.

    Expected: Every active alert is either resolved or explicitly accepted with a written reason.

  • TP-603 Handover artefacts As-built is complete E10 · Story 10.2
    1. Confirm the as-built records every FQDN, IP, VLAN and subnet actually deployed — the as-deployed state, not the plan.
    2. Confirm the deployment JSON specification and any stretch specification are stored with the engagement records.
    3. Confirm the network and connectivity model is documented, including the per-domain choices.
    4. Confirm the backup configuration, its target and its schedule are documented.

    Expected: The as-built reflects the deployed environment, with specifications and backup configuration stored alongside.

  • TP-604 Handover artefacts Every credential is in the secret store Critical E10 · Story 10.2
    1. Confirm all auto-generated component passwords from bring-up are stored (captured via Review Passwords at TP-106).
    2. Cross-check the inventory of what VCF manages against what you have stored, using tools/Get-VCFCredentials.ps1 from this repo: .\Get-VCFCredentials.ps1 -SDDCManager <fqdn> -Credential (Get-Credential) lists the accounts SDDC Manager holds for ESXi, vCenter, NSX and backup.
    3. Confirm the host root passwords, appliance shell credentials and the directory bind account are stored.
    4. Confirm the backup encryption passphrase is stored — and that it was actually proven to open an archive at TP-214.
    5. Where in scope, confirm the load-balancer controller PASSPHRASE (restore-critical) and the SSP Installer credentials are stored.
    6. Confirm the break-glass local account is stored, documented, and held out of band from the identity provider.
    7. Spot-check several entries by logging in with them right now. A stored credential that was mistyped is indistinguishable from a correct one until you try it.

    Expected: Every credential the environment depends on is in the secret store and spot-checked as working.

    There is no reveal API for VCF-managed passwords — management is rotate-and-expire only. Anything not captured here is unrecoverable without a rotation.

  • TP-605 Handover artefacts Test results are complete and evidenced Critical E10 · Story 10.3
    1. Confirm every in-scope case has a status, a date and an actual result.
    2. Confirm every pass has filed evidence — a screenshot, log excerpt or API response.
    3. Confirm every non-critical failure has a named owner and an agreed date.
    4. Confirm every not-applicable has a written reason.
    5. Confirm no critical failure remains open.

    Expected: All cases are executed and evidenced, no critical failure is open, and every non-critical failure and exclusion is owned and explained.

  • TP-606 ESX Security baseline is applied or its deviations are accepted E10 · Story 10.3
    1. Confirm the highest-priority hardening controls are applied or carry a documented, accepted deviation.
    2. Confirm the VCF Installer appliance was powered off or removed after bring-up — UNLESS it became SDDC Manager (the host-resident route), in which case it must stay: Get-VM | Where-Object Name -match "installer" | Select Name,PowerState
    3. Spot-check a few ESX controls against the baseline: Get-VMHost | Get-AdvancedSetting -Name Security.AccountUnlockTime,Security.AccountLockFailures,Security.PasswordHistory | Select @{n="Host";e={$_.Entity}},Name,Value
    4. Confirm every deviation records the parameter, the chosen value, the business reason, and any compensating control. Undocumented deviations are indistinguishable from misconfiguration at the next audit.

    Expected: Top-priority controls are compliant or carry an accepted, documented deviation, and the installer has been dealt with correctly.

    Sequence hardening against this plan: powering off the installer belongs after bring-up is verified, and disabling API basic authentication will break any tooling that still uses it.

  • TP-607 Handover artefacts Operations team walkthrough and sign-off Critical E10 · Story 10.3
    1. Walk the operations team through the platform: topology, the connectivity model per domain, where to look when something breaks.
    2. Walk through backup and restore, including where the passphrase lives.
    3. Walk through lifecycle: the depot, how updates are applied, and the certificate and licence expiry dates.
    4. Hand over the as-built and the completed test results.
    5. Obtain written sign-off.

    Expected: The operations team has been walked through the platform and its lifecycle, has the as-built and results, and has signed off.

Cases are written for this guide and cover the whole deployment, including the parts vendor verification material tends to leave out: the pre-bring-up readiness gate, distributed (Transit Gateway / VNA) connectivity, stretched clusters, backup verification, License Hub and load-balancer licensing, non-vSAN principal storage, and Supervisor beyond “the control-plane VMs booted”. Build the scope on the Deployment Plan Export page.