Portal

Product: Semarchy Data Platform

Version: 1.3.0 or later

Author: Hélène Zosym

Need

To deploy an SDP Self-Hosted in an air-gapped environment (without an outbound Internet access from Kubernetes cluster) it is required to first mirror the SDP helm chart and required images from the Semarchy Harbor repository to your local repository. While you can retrieve the list of the items to mirror manually, it's an error-prone process. This article describes how to automate the process end to end using the mirroring script and values overlay published in Semarchy's ps-public-artifacts repository, and how to wire that automation into a CI pipeline so it can be re-run for every new chart version. It is intended for infrastructure and platform teams preparing a self-hosted, air-gapped SDP deployment. 

Summarized Solution

• Authenticate to both the Semarchy source registry and your local target registry before running anything.
• Run mirror-semarchy-chart.sh, which pulls the chart, renders it with helm template to discover every image reference, copies each image with skopeo, and pushes the chart to your registry.
• Repoint the deployment values using values.customer-mirrored-images.yaml — three settings cover all twenty images: global.image.registry, plus two hard-coded overrides for dm.secretManager and reloader.
• Create an image pull secret in the target namespace and reference it from the values file.
• Install the platform from your local registry and verify with a helm template grep that no image still points at registry.na.semarchy.net.
• Wrap the whole flow in a CI pipeline (an Azure DevOps example is provided) so a new chart version can be mirrored on demand, with a --dry-run mode to review the plan first.

Detailed Solution

1. Deliverables and Prerequisites 

The repository provides two files that do all the work:

Required on the machine or CI runner that performs the mirroring:

On the target Kubernetes cluster: a namespace for the platform, and an image pull secret for the local registry (created in Step 4 below).

2. Running scripts to mirror charts and images

Step 1: Log in to Both Registries

The script assumes both logins already exist before it runs. Both are written to the Helm registry config, which the script then hands to skopeo via --authfile so images authenticate with the same credentials.

# Source (Semarchy) — credentials provided by Semarchy
helm registry login registry.na.semarchy.net \
  --username "$SEMARCHY_USER" --password "$SEMARCHY_KEY"

# Target (your local registry)
helm registry login artifactory.company.com \
  --username "$LOCAL_USER" --password "$LOCAL_KEY"

./mirror-semarchy-chart.sh \
  --target-repo   artifactory.company.com/semarchy \
  --chart-version 1.3.0

registry.na.semarchy.net/semarchy-release/docker-dm:1.12.1
  ->  artifactory.company.com/semarchy/semarchy-release/docker-dm:1.12.1

Run once with --dry-run to review the full plan — chart pull, every image copy, and the chart push — before mirroring for real.

global:
  image:
    registry: '<YOUR_LOCAL_REGISTRY>'   # same value as --target-repo above
    repository: 'semarchy-release'      # keep as-is — the mirror preserves this path

dm:
  secretManager:
    image:
      repository: '<YOUR_LOCAL_REGISTRY>/semarchy-release/docker-secret-manager'

reloader:
  image:
    repository: '<YOUR_LOCAL_REGISTRY>/semarchy-release/mirror/stakater/reloader'

Note: Set global.image.registry to the exact same base passed as --target-repo in Step 2 of the previous chapter so the paths line up. Also replace the remaining starter placeholders — <YOUR_GLOBAL_DOMAIN>, <YOUR_LICENSE_KEY_HERE>, <YOUR_INGRESS_CLASS_NAME>, the TLS secret names, the admin user fields, and <YOUR_LOCAL_REGISTRY_PULL_SECRET>.

Step 2: Create the Image Pull Secret

Installation now pulls from your registry, so create a pull secret in the release namespace and reference it — the values file already wires it via global.imagePullSecrets: 

kubectl create secret docker-registry local-registry \
  --docker-server=artifactory.company.com \
  --docker-username="$LOCAL_USER" \
  --docker-password="$LOCAL_KEY" \
  --namespace=semarchy

Use --docker-server=<host only> (no path). The secret name (local-registry here) must match global.imagePullSecrets[0].name in the values file. 

Step 3: Verify the Mirroring

Confirm no image reference still points at the source registry: 

helm template sdp \
  oci://artifactory.company.com/semarchy/semarchy-data-platform \
  --version 1.3.0 -f values.customer-mirrored-images.yaml \
  | grep -E '^\s*-?\s*image:' | sort -u

5. (Optional) Automating with CI

An Azure DevOps pipeline that performs the login and mirror steps is provided as azure-pipelines.yml. It runs manually (trigger: none), takes the chart version and target repo as run-time parameters, and pulls registry credentials from a secret variable group — nothing sensitive is stored in the pipeline file itself.

One-time setup

1. In Azure DevOps: Pipelines > Library > + Variable group, name it semarchy-mirror-creds.
2. Add SRC_HELM_USER, SRC_HELM_KEY, TGT_HELM_USER, TGT_HELM_KEY and mark each as secret.
3. Pipeline > Edit and link the semarchy-mirror-creds variable group.
4. Commit mirror-semarchy-chart.sh at the repo root (or adjust scriptPath).

At queue time, the pipeline prompts for:

The pipeline installs Helm and skopeo on an ubuntu-latest agent, logs in to both registries using a shared HELM_REGISTRY_CONFIG auth file, then invokes mirror-semarchy-chart.sh with the supplied parameters.

6. Upgrading to a New Chart Version

Image tags are tied to the chart version, so for every new release:

1. Re-run Step 2 of chapter 2 (or the CI pipeline) with the new --chart-version to mirror the new chart and any new or updated images.
2. Confirm the image set with the script's --dry-run output - if Semarchy adds or renames an image, it appears here first. 
3. Check that now images are flagged by Semarchy in release notes as requiring manual overwrite

7. Known Limitations and Troubleshooting

8. Mirroring Checklist