Reference

Binary Depot — Offline Depot & the VCF Download Tool

The binary depot the platform installs and patches from — online, offline (air-gapped), or manual transfer — plus the VCF Download Tool that populates it and the proxy the fleet may need to reach it. The prerequisites gate states what must exist; this page covers how to build and feed it. Companion page: SFTP Backup Target — the target every management component backs up to.

What the depot does not cover: SSP / License Hub. If vDefend or Avi is in scope, the Security Services Platform Installer (.ova) and the License Hub package (.tar) are downloaded by hand from the Broadcom Support Portal — roughly 9.5 GB between them — and the License Hub package is then uploaded to the SSP Installer. Neither passes through the Fleet Depot Service or the offline depot built here, which is VCF-component scoped. Plan that transfer separately, especially for an air-gapped site. See prerequisites.md → License Hub.

Contents

#SectionUse it when
1Three ways to feed binariesChoosing: online depot, offline depot, or manual transfer
2Setting up an offline depotThe site has no internet path to Broadcom
Step 1 — Depot web serverSizing and certifying the box
Step 2 — Auth splitWhat to protect with basic auth, and what must stay open
Step 3 — VCF Download ToolGetting the tool
Step 4 — Activation codeStart here early — the Product Administrator role takes days
Step 5 — Download the binariesThe actual binaries download run
Step 5b — ESX patch data (UMDS)esx download — the UMDS patch store, served without auth
Step 6 — Transfer to the air-gapped serverMoving the store across the gap intact
Step 7 — Connect VCF to itPointing the Installer (and later the fleet) at the depot
3Manual transfer — feeding the VCF Installer without a depot serverYou have no depot server at all and need bits on the Installer
4Using the Download Tool standalonePulling binaries without standing up a depot
5Proxy for the VCF services runtimeThe fleet has no direct internet — set the G5 proxy on the runtime via the Fleet LCM API (+ tools/ scripts)
Gotcha: precheck is a netcat test from the whole node blockPrecheck times out even with the documented access — firewall the whole services-runtime block
6Upgrades — filling the depot for a fleet upgradeThe fleet is already deployed and you are patching it
The loopSync → check → export → download → re-check
Two ways to get the binariesThe spec file, or a filtered catalog pull — and the size trade
Gotcha: a fresh download lands root-owned403 on a new file / precheck fails at Stage Precheck Binaries — re-apply perms, and the post-download hook
Reclaiming space: binaries cleanupThe depot is full — pruning superseded lines safely
7ReferencesThe TechDocs behind the above
AAppendix A — air-gapped Photon + nginx, end to endThe depot box is a minimal Photon image with no internet path

1. Three ways to feed binaries

Which path do I need?

Your situationRead firstAlso read
Site can reach Broadcom on 443Online depot (below)§5 if the fleet has no direct route
Air-gapped, building a depot others will reuse§2 end to end — plus Appendix A if the box is Photon§5 for the fleet proxy
One-off / lab install, no depot server to build§3 Manual transfer
Pre-staging binaries on a connected host, appliances stay offline§4
Fleet already deployed and you are patching it§6Step 5b for ESX patch data

Pick one feed method (intake G1):

  • Online depot — VCF Installer (and later the fleet) connect to the Broadcom depot directly using the Download Service ID + Activation Code (intake G2/G3; how to obtain them — and the Product-Administrator-role gotcha — is in §2 step 4). Needs outbound 443 to the Public URLs table (via the proxy from intake G5 if there is one — for the fleet’s own proxy, set on the VCF services runtime, see §5). TechDocs: Connect VCF Installer to Broadcom or an Offline Depot and Download Binaries.
  • Offline depot — for air-gapped sites the VCF Download Tool is the only supported method in 9.1. It downloads a depot store on an internet-connected host; your own web server serves that store to the VCF Installer and the fleet. The tool replaced the old Offline Bundle Transfer Utility (OBTU) and wraps UMDS for ESX patch data. TechDocs: Download Binaries to an Offline Depot by Using the VCF Download Tool.
  • Manual transfer — no depot at all: run the VCF Download Tool on any internet-connected machine, copy the depot store onto the VCF Installer appliance itself and import it there. No web server to build — best for a one-off install; see §3 for the steps and the Day-N caveat. TechDocs: Manually Transfer Binaries to VCF Installer.

2. Setting up an offline depot

The happy path, once you have done this before — every command is expanded in the like-numbered step below:

# 4  register: software depot ID -> activation code (VCF Business Services console)
./vcf-download-tool configuration generate --software-depot-id

# 5  pull the install set into the depot store
./vcf-download-tool binaries download --sku VCF --vcf-version 9.1.x \
  --depot-download-activation-code-file /path/activation-code.txt \
  --type INSTALL --depot-store /var/www/offline_depot

# 5b  pull ESX patch data (UMDS) into the same store
./vcf-download-tool esx download \
  --depot-download-activation-code-file /path/activation-code.txt \
  --depot-store /var/www/offline_depot

# after every pull (see §6): make the tree web-server-readable
chown -R nginx:nginx /var/www/offline_depot && chmod -R a+rX /var/www/offline_depot

# 6  if the download host is not the web server, move the store across intact
rsync -a /var/www/offline_depot/ depot01.sfo.example.io:/var/www/offline_depot/

# 7  point the VCF Installer at https://depot01.sfo.example.io/ with the basic-auth user

First time through — or when a step fails — read the detail below. Steps 1–3 (web server, auth split, get the tool) have no one-liner; do them once.

Step 1 — Depot web server

A Linux or Windows VM (any distribution), static IP (DNS record recommended), a dedicated disk for the depot store (sizing below), and any web server (Apache, NGINX) serving HTTPS with TLS 1.2/1.3. Give it a certificate with the FQDN and IP as SANs — signed by your CA, or self-signed if you accept the trust-import step (step 7 below).

Generating the cert (self-signed). The gotcha: modern clients — the VCF Installer included — ignore the CN and read only the subjectAltName, so a cert with just a CN is rejected even though it looks valid. Put the FQDN and the IP in the SANs:

mkdir -p /etc/nginx/ssl
openssl req -x509 -newkey rsa:2048 -nodes -days 825 \
  -keyout /etc/nginx/ssl/depot.key -out /etc/nginx/ssl/depot.crt \
  -subj "/CN=depot01.sfo.example.io" \
  -addext "subjectAltName=DNS:depot01.sfo.example.io,IP:10.11.10.20"
chmod 600 /etc/nginx/ssl/depot.key

-addext needs OpenSSL 1.1.1+ (Photon 4/5 both have it). For a CA-signed cert instead, swap -x509 … -out depot.crt for -new … -out depot.csr, hand the CSR to your CA, and drop the returned cert in as depot.crt. Once the cert is in place and referenced in the server block, load it with nginx -t && systemctl reload nginx (a running nginx re-reads its certs on reload; restart works too). A self-signed cert means the VCF Installer must trust it — see step 7.

HTTP depot (9.1) — skips the cert, but drops auth and the UI. VCF 9.1 added support for an offline depot served over plain HTTP (the VCF Installer and the Fleet Depot Service). It avoids the certificate entirely, but two things change: there is no authentication — an HTTP depot is anonymous (the Step 2 auth split exists only on HTTPS, as the Installer UI itself notes: “Authentication is supported only with an HTTPS offline depot”) — and the Installer UI will not register an HTTP depot; you must use the VCF Installer API. On nginx it is just listen 80; with no ssl_* and no auth_basic (open 80 instead of 443 in iptables). Only worth it on a locked-down segment where an unauthenticated depot is acceptable; otherwise stay on HTTPS + a self-signed cert + the step 7 trust-import. Ref: William Lam, New HTTP Offline Depot Support for VCF Installer & Fleet Depot Service.

Disk sizing. Start around 300 GB for the initial INSTALL depot (the bring-up bundles + component OVAs + ESX ISO). Broadcom’s own recommendation is to provision ≥ 1 TB, and that headroom is real rather than padding: the store grows every Day-N patch cycle — the fleet Depot Service side-loads too (see the §3 Day-N caveat) — on the order of a few GB per ESX patch pull. So build on ~300 GB, provision the 1 TB. No authoritative content footprint is published; 300 GB is a field starting point, not a hard minimum.

Ports. The only listening port the platform needs on the depot server is inbound TCP 443 (HTTPS) — the web server. The clients that pull from it are the same “Needed by” set as the Public URLs table, just pointed at your depot instead of Broadcom: the VCF Installer at bring-up, then the SDDC Manager / VCF Operations depot services runtime / vCenter for Day-N patching. Open 443 to the depot from the management network and the whole services-runtime block — the fleet Depot Service pulls from there, the same “firewall the block” lesson as the proxy in §5. No inbound 80 is required; serve HTTPS only (an 80 → 443 redirect is optional convenience, not a requirement). Outbound: the depot box needs none — unless it is also the internet-connected host running the VCF Download Tool, in which case it needs outbound TCP 443 to the Broadcom Public URLs. An air-gapped depot (store copied in) makes no outbound connections.

Depot box on Photon OS? A minimal, air-gapped Photon image hides real work the generic “install nginx, point it at the store” skips — offline tdnf, the nginx service user, htpasswd absent from the image, iptables persistence. Appendix A walks it end to end; the rest of §2 (cert SANs, auth split, connecting VCF) applies unchanged.

Step 2 — Auth split

Protect PROD/COMP and PROD/metadata with basic auth (htpasswd); leave PROD/vsan/hcl and umds-patch-store open. The umds-patch-store directory name is hardcoded — don’t rename it.

Step 3 — VCF Download Tool

Download it from the Broadcom Support Portal (My Downloads → VMware Cloud Foundation → your version → Drivers & Tools) and extract it on an internet-connected host — the depot server itself if it’s allowed out, otherwise any staging machine.

Step 4 — Activation code

From 9.1, whatever connects to Broadcom for binaries (VCF Installer, a software depot, the Download Tool) must be registered in the VCF Business Services console — the authoritative how-to is Software Depot Registration in the VCF Business Services Console (per-component registration sub-pages). For the Download Tool: generate a software depot ID with the tool (./vcf-download-tool configuration generate --software-depot-id), then log in to the console, select the tenant + site ID that map to your VCF entitlement, and generate the activation code against that depot ID. Save the code to a text file for the --depot-download-activation-code-file flag. (Code not appearing? — KB 399124.)

Token vs. activation code: Broadcom is mid-transition. The older download token (support portal → My Dashboard → Generate Download Token, see KB 390098) still works for 9.1 downloads — the 9.0-era tool took it as a download-token file — but the activation code is the go-forward mechanism that replaces the token workflow.

Get the credential early: generating either requires the Product Administrator role on the Broadcom support-portal site. If your named contact doesn’t have it, the site’s User Administrator must assign it first — plan days for this, not minutes.

Don’t confuse it with the licensing activation code. VCF 9 also uses an activation code to register VCF Operations with the licensing service (vcf.broadcom.com) for the single fleet license file — a separate credential from the depot-download one, generated in its own registration wizard. The activation/registration model is explained in the VMware blog Licensing in VMware Cloud Foundation 9.0 (Sehjung Hah & Kyle Gleed).

Step 5 — Download the binaries

Into the depot store (the web server’s document root, or a staging directory):

./vcf-download-tool binaries download --sku VCF --vcf-version 9.1.x \
  --depot-download-activation-code-file /path/activation-code.txt \
  --type INSTALL --depot-store /var/www/offline_depot

binaries list (same flags) previews what a run will pull; --type UPGRADE fetches lifecycle bundles for Day-N patching. This --type INSTALL set carries the ESX ISO (host imaging) but not ESX patch data — that is a separate pull, see Step 5b.

NSX Edge nodes need no extra binary. There is no separate NSX Edge bundle in the depot — the edge node OVA ships inside the NSX Manager appliance (NSX_T_MANAGER, already part of the --type INSTALL set), and NSX Manager deploys the edge VMs itself when you create an edge cluster after bring-up. The Broadcom edge-cluster prerequisites are network/resource-only (Edge TEPs, uplinks, DNS, BGP) and never mention the depot, and the depot component list confirms it (see William Lam’s depot structure deep-dive). An air-gapped depot loaded per this section therefore already covers a later edge-cluster deployment; Day-N, edge nodes are upgraded through the NSX upgrade bundle (--type UPGRADE), not a separate download.

Behind a proxy? Use the tool’s own flags — the shell env vars are ignored. http_proxy/https_proxy have no effect; the Download Tool reads only its own options. Add --proxy-server <FQDN:Port> (-s, no http:// scheme). The symptom without it is Fail to obtain access token from Broadcom OAuth Authorization server plus name-resolution errors — because with a proxy the proxy resolves the Broadcom names, not the depot box:

./vcf-download-tool binaries download --sku VCF --vcf-version 9.1.x \
  --depot-download-activation-code-file /path/activation-code.txt \
  --type INSTALL --depot-store /var/www/offline_depot \
  --proxy-server proxy01.sfo.example.io:3128

If the proxy is an HTTPS proxy, add --proxy-https — which first requires the proxy’s certificate imported into the tool’s JRE default trust store. If it authenticates, add --proxy-user <user> (-r) and --proxy-user-password-file <path>. (The depot … upload subcommands cannot use the proxy filter.)

Gotcha — Failed to get last updated time for HCL. Even once the proxy works the download can fail here: the tool queries vsanhealth.vmware.com for the vSAN HCL and chokes. Fix is an endpoint swap — change vsan.hcl.client.endpoint=vsanhealth.vmware.com to vsan.hcl.client.endpoint=eapi.broadcom.com in both conf/application-prod.properties and conf/application-prodV2.properties, then retry. Broadcom KB 438222.

(Re)apply the Step 2 security after the download. The binaries download creates and refreshes the depot’s PROD/… tree, so the Step 2 auth split has to be applied once the store is populated — not before — protecting PROD/COMP and PROD/metadata while leaving PROD/vsan/hcl and umds-patch-store open. Do it again after every Day-N --type UPGRADE refresh, since each run rewrites the tree. Re-verify with the curl checks (open paths return 200, protected paths prompt) and confirm the web-server user can still read the newly written files.

Step 5b — ESX patch data (UMDS)

Step 5’s --type INSTALL set includes the ESX ISO used to image and commission hosts. It does not include ESX patch data — the rollup bulletins vCenter’s vLCM needs to remediate running hosts Day-N. Pull that separately:

./vcf-download-tool esx download \
  --depot-download-activation-code-file /path/activation-code.txt \
  --depot-store /var/www/offline_depot

esx download wraps UMDS — it replaced the standalone UMDS install and the old Offline Bundle Transfer Utility (OBTU) — and mirrors the ESX patch depot into a umds-patch-store/ directory beside PROD/ under the document root. Four things bite here:

  • umds-patch-store is a hardcoded directory name — don’t rename it, and serve it without basic auth. The Step 2 auth split protects PROD/COMP and PROD/metadata only; umds-patch-store (like PROD/vsan/hcl) must stay anonymously reachable or vLCM cannot read the patch metadata.

  • Behind a proxy: same rule as Step 5 — add --proxy-server <FQDN:Port> (no http:// scheme); the shell http_proxy / https_proxy env vars are ignored.

  • Re-apply permissions after the run. esx download runs as root and writes the store root-owned, and it is not covered by the §6 post-download hook (that wraps only binaries download). Re-run the ownership fix by hand, or rely on the systemd path unit from §6 which catches any write:

    chown -R nginx:nginx /var/www/offline_depot
    chmod -R a+rX /var/www/offline_depot
  • Verify it is anonymously reachable once the web server is serving it:

    curl -k https://depot01.sfo.example.io/umds-patch-store/   # -> 200, no auth prompt

Day-N: re-run esx download whenever you want newer ESX patch lines in the depot, then re-apply permissions and re-run CHECK BINARY AVAILABILITY (§6) — that, not anything the CLI reports, is the authoritative check on whether the fleet can see the new data. ESX upgrade bundles (major host version upgrades) are a different pull — binaries download --type UPGRADE --component ESX_HOST, into PROD/COMP.

Step 6 — Transfer to the air-gapped server

Move the depot store to the air-gapped web server if the download host is a separate machine — the directory tree (PROD/COMP, PROD/metadata, PROD/vsan/hcl, umds-patch-store) must arrive intact under the document root.

Step 7 — Connect VCF to it

Point the VCF Installer at the depot URL (e.g. https://depot.sfo.rainpole.io/) with the basic-auth user (intake G4). The Installer must trust the depot’s TLS certificate — with a self-signed or internal-CA cert, plan the certificate import (in 9.0 there was no accept-certificate prompt; the cert had to be imported over SSH — see the vTam walkthrough below). Day-N, the fleet connects under VCF Operations → Depot Configuration (fleet-level and per-instance).

3. Manual transfer — feeding the VCF Installer without a depot server

For a one-off installation (lab, PoC, or a small air-gapped site) you can skip building the depot web server entirely: download the depot store with the VCF Download Tool on any internet-connected machine, copy it onto the VCF Installer appliance, and import it locally. “Manual” means no depot server — not hand-picked downloads: the binaries still come exclusively through the VCF Download Tool + activation code (§2 steps 3–4 apply unchanged); pulling OVAs/ISOs by hand from the support portal is not a supported substitute in 9.1.

  1. Download the install set on the internet-connected machine — §2 step 5’s binaries download --type INSTALL command, with --depot-store pointing at a local staging directory. Metadata comes with it; plan the same order of disk space.

  2. Copy to the Installer — put the Download Tool itself on the appliance (extract the .tar.gz under /nfs/vmware/vcf/nfs-mount/vcfdt) and transfer the staged store:

    rsync -aP /path/to/binaries vcf@installer-fqdn:/nfs/vmware/vcf/nfs-mount/depot
  3. Import on the appliance — save the admin@local password to a text file, then:

    ./vcf-download-tool binaries upload --depot-store /nfs/vmware/vcf/nfs-mount/depot \
      --sddc-manager-fqdn installer-fqdn --sddc-manager-user admin@local \
      --sddc-manager-user-password-file /path/to/password.txt

    Once the upload finishes, the Installer shows the binaries as available and deployment proceeds as normal — no depot configured anywhere.

Day-N caveat (new in 9.1). After bring-up the fleet runs its own Depot Service, and with no depot connected, patch/upgrade binaries must be side-loaded there too — a second upload per cycle (vcf-download-tool depot binaries upload --ops-fqdn <vcf-ops-fqdn> --depot-fqdn <fleet-depot-fqdn>, optionally per --component). In 9.0 this extra step didn’t exist. Manual copies at every patch cycle get tedious fast — beyond a lab or one-off, the offline depot (§2) is the smoother long-term setup. Walkthrough incl. the Fleet Depot Service step: William Lam’s Side-loading VCF binaries into VCF Installer & Fleet Depot Service.

4. Using the Download Tool standalone

You don’t need to be air-gapped to use the tool. Run it on any internet-connected machine with the same activation-code file to pre-stage binaries locally — for example to pull the --type INSTALL set before an installation window, or ESX patch data via its built-in UMDS, without letting the VCF appliances out to the internet. Whatever machine runs it needs outbound 443 to the Public URLs — plan that host’s egress (or proxy allowlist) as part of the prereq gate.

5. Proxy for the VCF services runtime (via the Fleet LCM API)

When the VCF Management Services components have no direct route to the internet, the fleet reaches the online depot (and everything else it downloads) through a proxy set on the VCF services runtime — the VSP component in Fleet lifecycle. This is the G5 proxy referenced in §1, applied to the fleet rather than the Installer. TechDocs: Configure a Proxy Server for VCF Management Services Components and VCF Automation.

The TechDocs procedure reads as a wall of curl/jq token-juggling; underneath it is short. It’s the same Fleet LCM API pattern as the backup config in Backup Target §5, so the field notes there apply here too.

The flow, distilled:

  1. Authenticate through VCF Operations (the only issuer the Fleet lifecycle service trusts):

    • POST https://<VCFOps>/suite-api/api/auth/token/acquire (username/password) → an OpsToken
    • POST https://<VCFOps>/suite-api/api/auth/token/exchange with header Authorization: OpsToken <token> and body {"serviceKeys":["fleet-lcm"]} → the Fleet LCM Bearer JWT
  2. Find the runtime: GET https://<FleetLCM>/fleet-lcm/v1/components → the component whose componentType is VSP, take its id.

  3. Set the proxy: PATCH https://<FleetLCM>/fleet-lcm/v1/components/<vspId>/config

    {
      "type": "VspClusterConfigSpec",
      "peerProxy": {
        "host": "10.11.10.250",
        "port": 3128,
        "tlsEnabled": false,
        "credentialsEnabled": false
      }
    }

    Returns a task id.

  4. Watch / verify: GET .../fleet-lcm/v1/tasks/<taskId>, then GET .../fleet-lcm/v1/components/<vspId>/config.

Optional peerProxy fields: username + password (with credentialsEnabled: true for an authenticating proxy), encodedCertificate (base64 PEM, with tlsEnabled: true for an HTTPS proxy), and excludeDomains / excludeIpAddresses for no-proxy bypasses.

Set the exclusions — a bare proxy sends everything out. With only a host and port, all of the services runtime’s HTTP egress is forced through the proxy, including traffic that must stay on the management network: the other VCF appliances (vCenter, NSX, SDDC Manager, VCF Operations) and — the one that bites in an air-gapped build — your on-prem offline depot. At best that adds a hop; at worst the proxy refuses to route the internal/RFC1918 destination and the connection fails. So set:

  • excludeDomains — your internal DNS suffix(es), e.g. sfo.example.io, so every internal FQDN bypasses the proxy.
  • excludeIpAddresses — the management / services-runtime CIDRs and the subnets holding vCenter/NSX/SDDC Manager/VCF Operations, plus the offline depot’s IP or subnet so bundle pulls stay internal.

With the scripts: -ExcludeDomains 'sfo.example.io' -ExcludeIpAddresses '10.11.0.0/16','<depot-ip>'. They ride the same merge PATCH, so re-running Set-VCFProxyConfig.ps1 adds them alongside the existing host/port; confirm with Get-VCFProxyConfig.ps1 (empty exclude lines mean everything is being proxied).

Three traps worth knowing (all cost time to discover):

  • Go straight to the fleet appliance, not the VCF Operations proxy route. The browser sends this to https://<VCFOps>/vcf-operations/plug/fleet-lcm/..., but that path is the user interface’s session-authenticated route (it works because the browser holds a JSESSIONID cookie). A Bearer-token client gets 405 on a PATCH there and HTML on a GET. Both the lookup and the write go to https://<FleetLCM>/fleet-lcm/v1/..., where the token is accepted — the same lesson as the backup write (Backup Target §5, and issue #150).
  • Skip the /casa/services service-key lookup. The TechDocs procedure fetches a service key from <VCFOps>/casa/services with a hard-coded Basic-auth header; you don’t need it. The fleet-lcm service key is just the literal string "fleet-lcm" in the exchange body (proven live by the backup scripts).
  • port is a number here. In VspClusterConfigSpec the proxy port is a JSON number (3128), unlike the SFTP backup payload where port is a string. Send the wrong type and validation rejects it.

The two scripts in tools/ do exactly this, reusing the backup scripts’ auth chain — download them straight from the site:

ScriptWhat it does
Get-VCFProxyConfig.ps1Read-only. Shows the peerProxy the platform actually stored on each VSP (VCF services runtime) component. Changes nothing
Set-VCFProxyConfig.ps1Sets the proxy through the API. -WhatIf prints the exact payload (secrets masked) without sending it; supports an authenticating proxy (-ProxyUsername), a TLS proxy (-CertificateFile), and -ExcludeDomains / -ExcludeIpAddresses; -Remove clears it by sending explicit empty values (blank host, port 0) — the Fleet LCM PATCH is a merge, so a null peerProxy is a silent no-op (task completes, nothing changes). Verify with Get-VCFProxyConfig.ps1 afterwards

Gotcha: the precheck is a netcat test from the whole node block — even when the documented access is in place

Setting the proxy is not the end of it. The PATCH is accepted (you get a task ID), and the platform then runs a peer-proxy-precheck workflow on the VCF services runtime. If that precheck fails, the proxy is never applied — the config submit stores nothing, exactly like the backup submit in Backup Target §5. And the failure the platform surfaces is misleading: “the proxy server may be slow, overloaded, or network latency is too high.”

What the precheck actually does — read it yourself. Get a kubectl session on the control-plane node (recipe in Backup Target §7 — SSH in as root, kubectl is already wired up), then:

kubectl -n vmsp-platform get pods | grep peer-proxy-precheck        # the failing Error pods
kubectl -n vmsp-platform logs <peer-proxy-precheck-...-main-pod> -c main --tail=200

The log shows it is a plain L4 nc (netcat) TCP connect to the proxy host:port — no auth, no TLS, pure reachability:

Testing TCP port connectivity/reachability using nc (netcat)
nc: connect to 10.11.10.250 port 3128 (tcp) failed: Connection timed out
Proxy connectivity test failed: 10.11.10.250:3128

The gotcha: that nc runs from a pod that can be scheduled on any VCF services-runtime node, and its egress SNATs to that node’s IP. So the source of the proxy connection is the entire services-runtime node blocknot the depot or VCF Operations IPs. Broadcom’s documented access for the proxy does not call this out, so the firewall gets opened only for the depot + Ops IPs (the hosts you’d expect to use a download proxy), the precheck lands on a node that isn’t permitted, and it times out. This is the same trap as the backup target in Backup Target §5: firewall the block, not the named hosts you think talk to it.

Fix: allow the whole services-runtime node block outbound to the proxy on its port (e.g. TCP 3128). List the exact source IPs to hand the network team with:

kubectl get nodes -o wide      # the INTERNAL-IP column = the source IPs to allow

Then re-submit (Set-VCFProxyConfig.ps1 again, or let the platform retry) and the nc check passes. Because it is L4-only, an authenticating (credentialsEnabled) or TLS (tlsEnabled) proxy still has to clear this reachability gate first — fix the firewall before chasing credentials or certificates.

6. Upgrades — filling the depot for a fleet upgrade

Everything above is about install binaries. Once the fleet is running, patching it draws on a different set — and the UI does not tell you how to get them. In VCF Operations, Build → Lifecycle → VCF Management → Upgrade → CHECK BINARY AVAILABILITY offers EXPORT DOWNLOAD SPECIFICATION, which hands you a manage-binaries.json and the sentence “use it with the VCF Download Tool to obtain these binaries”. It never names the flag. It is --download-spec-file [field-verified].

The loop: sync → check → export → download → re-check

  1. Sync the lifecycle metadata first. The upgrade page shows Last lifecycle metadata sync time with a Sync link. If it reads N/A, sync it and confirm you get a real timestamp before going further — binary availability is evaluated against that metadata.

    Suspected, not proven. A specification exported while the sync read N/A produced an empty result from the Download Tool. That is consistent with the spec being generated from the metadata, but we did not inspect the JSON to confirm it. Treat “sync first” as cheap insurance, and if a download returns nothing, open the spec file and see whether it actually lists components: python3 -m json.tool manage-binaries.json | head -60.

  2. CHECK BINARY AVAILABILITY — tells you what the fleet needs and what the depot is missing. This is the authoritative answer, not anything the CLI reports.

  3. Export the download specification if you want the exact subset.

  4. Download (below), into the depot store the fleet is actually registered against — easy to fill a staging directory nobody reads.

  5. Re-run CHECK BINARY AVAILABILITY. If it still reports gaps after a successful download, suspect the metadata sync or the depot path, not the binaries.

Two ways to get the binaries

A — the specification file. Exactly the components for your hop, no version guessing:

./vcf-download-tool binaries download \
  --download-spec-file /root/manage-binaries.json \
  --depot-download-activation-code-file /root/reg.txt \
  --depot-store /depotdata \
  --proxy-server <fqdn:port>

B — a filtered catalog pull. No spec file, but you must filter well or you will pull far more than you need:

./vcf-download-tool binaries download --sku VCF --vcf-version 9.1.0 \
  --type UPGRADE --depot-store /depotdata \
  --depot-download-activation-code-file /root/reg.txt \
  --proxy-server <fqdn:port>

The size trade is not small. A bare release line returns every patch line at once9.1.0.0, .0100, .0200, .0300, .0400 — so a single point upgrade drags in four vCenter builds at 28.7 GiB each, three VCF Automation at 14.9 GiB, three NSX at 6.4 GiB. Comfortably over 100 GiB for a 0300 → 0400 hop [field-verified]. Filter, or use the spec.

C — by binary ID. The middle ground: no spec file, but no superfluous lines either. List first, take the IDs for the release you want, feed them back:

./vcf-download-tool binaries download \
  --id=<id1>,<id2>,<id3> \
  --depot-store /depotdata \
  --depot-download-activation-code-file /root/reg.txt \
  --proxy-server <fqdn:port>

The filters, from --help [field-verified]:

FilterNotes
--vcf-versionFormats a.b, a.b.c, a.b.c.d, and ranges (9.1.0..9.1.1, or 9.1.0.. for everything higher). Short forms match broadly9.1.0 returns 9.1.0.0 and every 9.1.0.x maintenance release. 9.1.0.0400 matches that release exactly
--idOne or more binary IDs, comma-separated or by repeating the flag. IDs come from the ID column of binaries list — the surgical option
--componentVCENTER, SDDC_MANAGER_VCF, NSX_T_MANAGER, ESX_HOST, VRA, VROPS, VRLI, VRNI, VSAN_FILE_SERVICES, VCF_FLEET_LCM, VCF_SDDC_LCM, VSP, DEPOT_SERVICE, HCX, …
--component-versionNarrow a single component
--typeINSTALL or UPGRADE — there is no PATCH type to pass, though UPGRADE returns rows displayed as PATCH
--lifecycle-managed-bySDDC_MANAGER_VCF, VRSLCM, VCF_FLEET_LCM, SELF
--upgrades-only / --patches-onlyAdditional narrowing

binaries list queries Broadcom’s catalog, not your depot. It needs the activation code and the proxy, and it tells you nothing about what you already hold locally. For local content, look at --depot-store on disk — and for “do I have what I need”, use CHECK BINARY AVAILABILITY. [field-verified]

Run long pulls detached — screen, or nohup … & — and watch <toolroot>/log/vdt.log. The tool runs its own free-space precheck and refuses rather than filling the disk.

Gotcha: a fresh download lands root-owned — nginx 403s and the precheck fails

Re-apply ownership and permissions after every download, not just at build time. The Download Tool runs as root and writes the new bundles root-owned; nginx runs as the nginx user, cannot read them, and answers 403. The chown/chmod in §2 Step 1 covers the store the day you build it — every subsequent pull re-introduces the problem. [field-verified]

chown -R nginx:nginx /var/www/offline_depot
chmod -R a+rX /var/www/offline_depot

a+rX is the form you want: capital X sets execute on directories only, so directories become traversable and files become readable, without marking bundle files executable. (Lower-case x would set it on everything.)

The symptom is nowhere near the filesystem. The fleet does not surface a 403. A VCF Operations upgrade precheck fails at subtask Stage Precheck BinariespreCheckBinaryOnly=true, component type OPS — with failed stages download_bundle_file, stage_bundle_file and a generic ops.task.stage.failed. That task’s only job is fetching bits, so a failure there is a binary-delivery problem, not an environment-readiness one. Depot file permissions are on the short list of causes, alongside the metadata sync, depot reachability from the whole services-runtime block (§5), and depot credentials.

Is it permissions or is it auth? /PROD/COMP and /PROD/metadata sit behind basic auth (§2 Step 2), so a 403 there is ambiguous. Separate the two by status code:

curl -k -u depotuser https://depot01.sfo.example.io/PROD/COMP/<new-file> \
  -o /dev/null -w '%{http_code}\n'
# 401 -> credentials / .htpasswd    403 -> filesystem permissions    200 -> fine

Check the whole path, not just the store root. nginx needs +x on every parent directory. If the depot store is on a dedicated disk, a 700 mountpoint blocks traversal even when everything beneath it is 755:

namei -l /var/www/offline_depot/PROD/COMP/<new-file>

That prints each path component with its mode and owner, so the first line that lacks x for nginx is your answer.

The durable fix: a post-download hook

Re-running two commands by hand works until the one time somebody forgets — and the failure surfaces days later as a precheck error that looks like a vROps problem. Automate it instead. Two options; the systemd one is what I would deploy.

Option A — systemd path unit (recommended). Watches the store and fixes permissions whenever it changes, regardless of who ran the download or how.

Create the oneshot service, /etc/systemd/system/depot-perms.service:

[Unit]
Description=Re-apply nginx-readable ownership/permissions to the VCF offline depot store

[Service]
Type=oneshot
ExecStart=/usr/bin/chown -R nginx:nginx /var/www/offline_depot
ExecStart=/usr/bin/chmod -R a+rX /var/www/offline_depot

And the path unit that triggers it, /etc/systemd/system/depot-perms.path:

[Unit]
Description=Watch the VCF offline depot store for new binaries

[Path]
PathChanged=/var/www/offline_depot
Unit=depot-perms.service

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now depot-perms.path
systemctl status depot-perms.path        # Active: waiting

Verify by touching a file in the store and confirming the service ran:

touch /var/www/offline_depot/.permcheck
systemctl status depot-perms.service     # should show a recent successful run
ls -l /var/www/offline_depot/.permcheck  # nginx:nginx
rm /var/www/offline_depot/.permcheck

Two caveats worth knowing before you rely on it. PathChanged fires on changes to the named directory itself, not recursively into subtrees — but since a full chown -R runs on every trigger, the tree still converges; you just may not get a trigger for a write deep inside an existing subdirectory until something touches a watched level. And it fires while a long download is still writing, which is harmless (the -R simply runs again), but it means you should still run the two commands once manually after a large pull completes rather than assuming the last trigger caught the final file.

Option B — wrap the download. Simpler, no systemd, but only covers downloads that go through the wrapper. Drop this next to the tool as depot-pull.sh:

#!/bin/sh
set -e
DEPOT_STORE=/var/www/offline_depot
/root/vcf-download-tool/vcf-download-tool binaries download \
  --depot-store "$DEPOT_STORE" "$@"
chown -R nginx:nginx "$DEPOT_STORE"
chmod -R a+rX "$DEPOT_STORE"
echo "Permissions re-applied to $DEPOT_STORE"
chmod 750 /root/depot-pull.sh
./depot-pull.sh --download-spec-file /root/manage-binaries.json \
  --depot-download-activation-code-file /root/reg.txt --proxy-server <fqdn:port>

set -e matters: if the download fails, you do not want the success message. Note this only catches binaries download — an esx download or a manual scp of the store across the air gap (§2 Step 6) bypasses it entirely, which is exactly why Option A is the better bet on a depot that several people touch.

Whichever you pick, re-run CHECK BINARY AVAILABILITY afterwards — it is the authoritative answer on whether the fleet can actually see the new bundles.

Reclaiming space: binaries cleanup

Same three filter groups as download[VCF VERSION] | [BUNDLE ID] | [DOWNLOAD SPEC] — and they are mutually exclusive. Documented examples, verbatim:

# install binaries for a version
sh vcf-download-tool binaries cleanup --depot-store=<binaries_dir> \
  --vcf-version=<VCF_version> --automated-install --sku=VCF --type=INSTALL

# upgrade binaries managed by SDDC Manager
vcf-download-tool binaries cleanup --depot-store=<binaries_dir> \
  --vcf-version=<VCF_version> --lifecycle-managed-by=SDDC_MANAGER_VCF --type=UPGRADE

# whatever a specification file describes
vcf-download-tool binaries cleanup --depot-store=<binaries_dir> \
  --download-spec-file=<download_spec_file_path>

Prune a superseded patch line — note the exact four-part version:

./vcf-download-tool binaries cleanup --depot-store=/depotdata \
  --vcf-version=9.1.0.0100 --type=UPGRADE

Delete specific artifacts by ID:

./vcf-download-tool binaries cleanup --depot-store=/depotdata \
  --id=1ed5da8d-...-3550bc51,0be4788c-...-2aadccc2

Get the IDs from a listing filtered the same way you intend to delete — each component has a different ID per type, so the VCENTER INSTALL row and the VCENTER UPGRADE row are different IDs.

Three cautions [field-verified]:

  • A short --vcf-version matches broadly. On download that costs bandwidth; on cleanup it deletes far more than you meant. Always use the exact a.b.c.d form when pruning one line.
  • There is no --dry-run. Preview by running binaries list with the identical filters first — imperfect, because list reads the catalog rather than your depot, but it shows what the filter matches.
  • Never clean while a download is writing to the same store.

What is safe to remove, in order of confidence: superseded patch lines you will not roll back to; INSTALL bundles for components already deployed (a running fleet upgrades from UPGRADE bundles) — but keep the ESX install bundle if you will ever commission a host or add a cluster.


7. References


Appendix A — air-gapped Photon + nginx, end to end

Photon is a natural pick for the depot box — lightweight and VMware-native — but it is a minimal image, and it is usually the air-gapped box, so Step 1’s “tdnf install nginx, point a web server at the store” needs spelling out. This appendix is that detail; the rest of §2 — cert SANs, the auth split, connecting VCF — is the same as any other build. End to end:

1. Install nginx — offline. The box typically has no internet, so tdnf needs a local source. Two ways, no internet required either way:

  • From the Photon ISO (no second machine). Photon ships a preconfigured photon-iso repo pointing at file:///mnt/cdrom/RPMS. Mount the full ISO the appliance was built from (the minimal ISO may not carry nginx), refresh the cache, and install from it alone:

    mkdir -p /mnt/cdrom && mount /dev/cdrom /mnt/cdrom   # or: mount -o loop photon-full-<ver>.iso /mnt/cdrom
    tdnf makecache
    tdnf install --disablerepo=* --enablerepo=photon-iso nginx
  • Or pre-download on an online Photon of the identical version/arch, copy the RPMs across (scp / USB), and install locally — deps resolve among them:

    # online box:      tdnf install --downloadonly --downloaddir=/root/nginx-rpms nginx
    # air-gapped box:  cd /root/nginx-rpms && tdnf install ./*.rpm

    (If the box can reach the proxy from §5, set proxy= in /etc/tdnf/tdnf.conf and just tdnf install nginx.)

2. Create the store directory — this is what you serve, and what the Download Tool writes into (pass it as --depot-store in Step 5):

mkdir -p /var/www/offline_depot
chown -R nginx:nginx /var/www/offline_depot && chmod -R a+rX /var/www/offline_depot

nginx runs as the nginx user, so it must be able to read the tree; Photon has no SELinux, so there is no doc-root labeling step you would hit on RHEL.

This is not a one-time step. Every later binaries download writes root-owned files into the store and re-breaks it — see §6 Gotcha: a fresh download lands root-owned for the symptom (a VCF Ops precheck failing at Stage Precheck Binaries) and a post-download hook that keeps it fixed.

3. Serve it over HTTPS with the Step 2 auth split. Add a server block to the http { } of /etc/nginx/nginx.conf (or /etc/nginx/conf.d/depot.conf if it includes conf.d):

server {
    listen 443 ssl;
    server_name depot01.sfo.example.io;        # must be in the cert SANs
    ssl_certificate     /etc/nginx/ssl/depot.crt;
    ssl_certificate_key /etc/nginx/ssl/depot.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    root      /var/www/offline_depot;          # the --depot-store dir
    autoindex on;                              # directory listing — handy to verify the tree

    # Step 2 auth split: protect COMP + metadata; HCL + UMDS stay open
    location /PROD/COMP     { auth_basic "VCF Depot"; auth_basic_user_file /etc/nginx/.htpasswd; }
    location /PROD/metadata { auth_basic "VCF Depot"; auth_basic_user_file /etc/nginx/.htpasswd; }
    # /PROD/vsan/hcl and /umds-patch-store have no auth_basic -> open, as required
}

Create the basic-auth user without htpasswd (it ships in httpd-tools, absent on the minimal image) — generate the hash with openssl and write the line:

printf 'depotuser:%s\n' "$(openssl passwd -apr1)" > /etc/nginx/.htpasswd
chmod 640 /etc/nginx/.htpasswd && chown root:nginx /etc/nginx/.htpasswd

4. Open inbound 443 in iptables. Photon’s default firewall allows only SSH (22) inbound and drops ICMP, so the rule must be added and persisted or it is lost on reboot:

iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables-save > /etc/systemd/scripts/ip4save   # Photon's iptables persistence file
systemctl restart iptables

(Add iptables -A INPUT -p icmp -j ACCEPT before saving if you also want the box to answer ping for reachability tests — see the ICMP note in Backup Target §3.)

5. Start + verify:

nginx -t && systemctl enable --now nginx
curl -k https://depot01.sfo.example.io/PROD/vsan/hcl/            # open      -> 200
curl -k -u depotuser https://depot01.sfo.example.io/PROD/COMP/  # protected -> prompts