Shutdown and Startup — Ordered Runbook

Powering a VCF 9.1 fleet down and back up is an ordered procedure, not a set of power operations. Broadcom states it plainly:

“Shutting down VMware Cloud Foundation, for example, during hardware or power maintenance, and then starting it up must be done in a way that prevents data loss or appliance malfunction, and supports collection of troubleshooting data. You follow a strict order and steps for shutdown and startup of the VMware Cloud Foundation management components.” [documented]

This page is the ordered sequence in one place, with the rules that are easy to miss — the fleet-level VCF Operations constraint, what shared NSX does to the order, and the infrastructure VMs that must go last.

Why it lives in a planning repo: your shutdown order is a function of what you deployed. Every optional component from 05-day2-deployments.md — VCF Automation, Operations for Networks, Cloud Proxy, License Server, VCF Management Services — is a step in the sequence below. Steps you never deployed simply drop out. Decide the fleet at planning time and this runbook writes itself; discover it during a power maintenance window and you are reading TechDocs at 02:00.

Contents

The full sequence, start to finish

The whole page is the ordered procedure; this is its shape. Shutdown and startup are mirror images — startup reverses the shutdown order.

1. Pre-checks (§1) — fleet healthy, backups current, and note what you deployed: your component set determines the order.
2. Learn the three ordering rules (§2) — VCF Operations is last out / first in at the fleet level; shared NSX moves with the first workload domain; infrastructure VMs go last.
3. Shut down the workload domains (§3) — all of them, first.
4. Shut down the management domain (§4) — the strict 11-step sequence.
5. Maintenance window.
6. Start the management domain (§5) — the reverse of step 4.
7. Start the workload domains (§6).

1. Before you touch anything

Before shutdown [documented]:

• “Verify that you have complete backups of all management components” — see 08-backup-target.md
• “Verify that the management virtual machines are not running on snapshots”
• VADP-based backup solutions shut down per vendor guidance
• Migrate the management vCenter to the first ESX host in the default management cluster — Broadcom gives this as a way “to reduce startup time”. Cheap to do, and it means you know which host to power on first
• Know which cluster hosts the management vCenter: “This cluster must be shut down last”

Before startup [documented]:

• External services available — “Storage, Active Directory, DNS, NTP, SMTP, and FTP or SFTP”. If AD/DNS/NTP run as VMs inside the management domain, see rule 3 below; you are about to need them before they exist
• VADP backup solutions “properly started and operational according to the vendor guidance”

2. The three ordering rules that catch people

Rule 1 — VCF Operations is last out, first in. At the fleet level.

“The VCF Instance that is running VCF Operations must be the last instance to shut down.” [documented]

and on the way back up:

“If you have shut down all VCF Instances in a VCF fleet, you must first start the VCF Instance, which is running VCF Operations.” [documented]

This is a constraint between VCF instances, not within one. Shut down instances that do not run VCF Operations and VCF Automation first. Multi-instance fleets get this wrong because each instance looks self-contained.

Rule 2 — shared NSX moves with the first workload domain.

“If the NSX Manager cluster and NSX Edge cluster are shared with other workload domains, shut down the NSX Manager and NSX Edge clusters as part of the shutdown of the first workload domain.” [documented]

Startup mirrors it — start the other workload domains first, then the one running the shared NSX Edge/VNA nodes, then the workloads that depend on NSX services [documented].

Rule 3 — infrastructure VMs go last.

“If the management domain contains virtual machines that are running infrastructure services like Active Directory, NTP, DNS and DHCP servers, stop these virtual machines last.” [documented]

And for vSAN, the wizard needs you to say so explicitly:

“For a vSAN cluster, select I confirm all VMs below are infrastructure VMs in the vSAN shutdown wizard.” [documented]

This is a planning decision, not just an operational one. If AD, DNS and NTP live inside the management domain you are shutting down, startup has a chicken-and-egg problem — the platform wants those services before the VMs providing them exist. Worth settling in 01-network-dns-plan.md.

3. Shutdown — workload domains first

“You shut down the customer workloads and the management components for the workload domains before you shut down the components for the management domain.” [documented]

Per workload domain, in order [documented]:

4. Shutdown — the management domain (11 steps)

The documented order [documented]. Steps for components you never deployed drop out — see 05-day2-deployments.md for which of these are optional.

Step 5 — VCF Management Services is the one this repo already documents in detail: 08-backup-target.md §7. The documented procedure [documented] is to find the control-plane node IPs in VCF Operations (Build → Lifecycle VCF Management Components → the VCF services runtime component → the Nodes table), download vcf_services_runtime_shutdown.sh from Broadcom KB 440874 onto the SDDC Manager appliance, chmod +x it, run it --dry-run first, then for real, and finally shut down the guest OS of the VMs the script lists. Do not hand-stop the components — the runtime has its own internal order.

Broadcom’s KB and TechDocs page give no run-time estimate for the script itself — budget it as an unknown until you’ve timed it once in your own environment. One field note worth carrying in regardless: Ward Vissers’ walkthrough of the same script flags powering off any VCF Automation VMs before running it, “to avoid interference with automation tasks” — consistent with VCF Automation already being step 1 in the table above, but worth confirming explicitly rather than assuming the ordering alone covers it.

Step 6 — VCF Operations has a step people skip [documented]:

1. VCF Operations admin UI at https://<vcf_operations_fqdn>/admin as the local admin user
2. System status → Take cluster offline, give a reason, OK.

   “This operation might take about an hour to complete.” Budget for it.

3. Then shut down the guest OS of each appliance in vCenter, “by following the order in Broadcom knowledge base article 341964” — the appliance order within the cluster matters and lives in that KB. TechDocs times this step too: shutting down the guest OS of each appliance individually [documented] “takes several minutes to complete” — small per-appliance, but it multiplies across however many nodes the VCF Operations cluster has.

Step 11 — hosts and vCenter. For a vSAN cluster the documented path is the vSAN Shutdown cluster wizard [documented]: verify vSAN Skyline health and that resync has finished, add root to the Exception Users list on any host in lockdown mode, then right-click the cluster → vSAN → Shutdown cluster, and confirm all pre-checks are green. Repeat for the other vSAN clusters, leaving the cluster hosting the management vCenter until last.

“Connection to vCenter is lost because the vSAN shutdown cluster wizard shuts it down. The shutdown operation is complete after all ESXi hosts are stopped.” [documented]

Note the orchestration host the wizard names before you lose the UI — you will want it on the way back up.

Non-vSAN clusters follow the separate Shut Down ESX Hosts with NFS or Fibre Channel Storage path instead.

5. Startup — the management domain

“You start the management components for the management domain first. Then, you start the management components for the workload domains and the customer workloads.” [documented]

The exact reverse of §4 [documented]:

Step 3 — NSX Manager. Power on the nodes in the vSphere Client, then log in to NSX Manager and confirm System → Configuration → Appliances shows the cluster Stable with all nodes available before moving on [documented].

“This operation takes several minutes to complete until the NSX Manager cluster becomes fully operational again and its user interface — accessible.” [documented]

Step 6 — VCF Operations is the reverse of the shutdown [documented]: power on the appliances “by following the order in Broadcom knowledge base article 341964”, then in the admin UI at /admin, System status → Bring Cluster Online.

“Bringing the cluster online might take about an hour to complete.” [documented] Two of these hour-long waits — one on the way down, one on the way up — belong in your maintenance-window estimate.

Step 7 — VCF Management Services has the single longest documented wait in the whole startup sequence [documented], per Start VCF Management Services: control-node startup “takes several minutes to complete”, but “worker nodes startup takes about 20 minutes to complete” — and “the startup operation might redeploy some of the worker nodes”, so don’t take the appliances powering on in vCenter as done; confirm in VCF Operations that all worker nodes are running and none are still being (re)deployed before moving on to the next step.

6. Startup — workload domains

Per workload domain [documented]:

Restart Clusters has no equivalent on the shutdown side, which is why it gets skipped:

“After you start the ESX hosts and the vCenter appliance for a workload domain, restart all clusters to re-initiate the integration with the principal storage.” [documented]

And rule 2 again, for shared NSX [documented]: start the other workload domains, then the one running the shared NSX Edge/VNA nodes, then the customer workloads that rely on NSX services.

7. Timing — what to budget the window for

Broadcom only publishes wait times for a handful of steps. Everything else in the sequence has to be timed in your own environment — host count, storage type, and cluster size all move the real numbers, and no vendor page or blog gives a generic formula for that. What follows is what’s actually documented.

A few things to carry into a change-window estimate specifically:

• The two VCF Operations cluster waits are the biggest known-fixed costs — roughly an hour going offline, roughly an hour coming back online, and they are not overlappable with anything else since rule 1 puts VCF Operations last out / first in at the fleet level. That’s ~2 hours before you even count host power-cycle time.
• On the way back up, VCF Management Services’ ~20-minute worker-node wait is the next biggest fixed cost, and it lands after the two VCF Operations hours in the startup order (step 7 in the table above) — so the hour-plus-hour-plus-20-minutes stacks sequentially rather than overlapping.
• Everything without a documented number should get timed once, deliberately, outside a real maintenance window — a dry run of the services-runtime script, and a dry run (or at least a stopwatch) on the vSAN shutdown/restart wizard against your actual cluster — so the first time you rely on a duration estimate isn’t during the maintenance window itself.

8. References

• Shutdown and Startup of VMware Cloud Foundation — the parent page
• Shutting Down VMware Cloud Foundation · Shut Down the Management Domain · Shut Down a Workload Domain
• Starting Up VMware Cloud Foundation · Start the Management Domain · Start a Workload Domain
• Shut Down vSAN and the ESX Hosts in the Management Domain · Shut Down the vSAN Cluster Using the Shutdown Cluster Wizard
• Shut Down VCF Management Services · Shut Down VCF Operations · Start the NSX Manager Nodes · Start VCF Management Services — source of the control-node / worker-node startup timings in §7
• Broadcom KB 341964 — the appliance order within the VCF Operations cluster (referenced by both the shutdown and startup pages)
• Broadcom KB 440874 — vcf_services_runtime_shutdown.sh, covered in detail in 08-backup-target.md §7
• Ward Vissers, VCF 9.1 How to Safely Shut Down the Services Runtime Cluster Using PowerShell — field note on powering off VCF Automation VMs before running the services-runtime shutdown script, cited in §4