Portal

export NAMESPACE=<SDP_NAMESPACE>
export RELEASE=<HELM_RELEASE_NAME>          # for example: sdp
export CLIENT_NS=<NAMESPACE_FOR_CLIENT_PODS> # for example: default

# PostgreSQL administrator (the account you used to prepare the databases)
export POSTGRES_HOST=<POSTGRES_HOST>
export POSTGRES_PORT=5432
export POSTGRES_ADMIN_USER=<POSTGRES_ADMIN_USER>
export POSTGRES_ADMIN_PASSWORD=<POSTGRES_ADMIN_PASSWORD>

# Search engine administrator
export SEARCH_ENGINE_TYPE=<opensearch|elasticsearch>   # same value as global.searchEngine.type
export SEARCH_ENGINE_URL=https://<SEARCH_ENGINE_HOST>:<PORT>
export SEARCH_ADMIN_USER=<SEARCH_ENGINE_ADMIN_USER>
export SEARCH_ADMIN_PASSWORD=<SEARCH_ENGINE_ADMIN_PASSWORD>

for s in semarchy-harbor keycloak-postgres dm-postgres dm-postgres-datasource-1 dm-postgres-datasource-2 \
         kafka-keycloak dm-kafka "${SEARCH_ENGINE_TYPE}-provider" mail-secret; do
  kubectl get secret "$s" -n "$NAMESPACE" -o json \
    | jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, .metadata.annotations, .metadata.labels)'
done > sdp-input-secrets.json

kubectl run psql-client --rm -it --image=postgres:16 -n "$CLIENT_NS" \
  --env="PGHOST=$POSTGRES_HOST" \
  --env="PGPORT=$POSTGRES_PORT" \
  --env="PGUSER=$POSTGRES_ADMIN_USER" \
  --env="PGPASSWORD=$POSTGRES_ADMIN_PASSWORD" \
  -- bash

psql postgres

DROP DATABASE "keycloak" WITH (FORCE);
CREATE DATABASE "keycloak" ENCODING = 'UTF8' LC_COLLATE = 'C' LC_CTYPE = 'C' TEMPLATE template0 CONNECTION LIMIT = -1;
ALTER DATABASE "keycloak" OWNER TO "keycloak";
GRANT ALL PRIVILEGES ON DATABASE "keycloak" TO "keycloak";
\c keycloak
GRANT ALL ON SCHEMA public TO keycloak;
\c postgres

Keep the pod open if you also need step 5. The keycloak role and its password are kept, so the keycloak-postgres Secret does not change.

DROP SCHEMA IF EXISTS  "selfhosted-dm-repo-user" CASCADE;

-- Repository schema and read-only access
CREATE SCHEMA "selfhosted-dm-repo-user";
ALTER SCHEMA "selfhosted-dm-repo-user" OWNER TO "selfhosted-dm-repo-user";
GRANT CONNECT ON DATABASE "selfhosted-dm" TO "selfhosted-dm-repo-user-ro";
GRANT USAGE ON SCHEMA "selfhosted-dm-repo-user" TO "selfhosted-dm-repo-user-ro";

\q
exit

kubectl run curl-client --rm -it --image=curlimages/curl -n "$CLIENT_NS" \
  --env="SEARCH_ENGINE_URL=${SEARCH_ENGINE_URL}" \
  --env="SEARCH_ADMIN_USER=${SEARCH_ADMIN_USER}" \
  --env="SEARCH_ADMIN_PASSWORD=${SEARCH_ADMIN_PASSWORD}" \
  --env="RELEASE=${RELEASE}" \
  -- sh

AUTH="${SEARCH_ADMIN_USER}:${SEARCH_ADMIN_PASSWORD}"
INDEXES="${RELEASE}-log-explorer-index-*,${RELEASE}-billing-metrics"

# List
curl -s -u "$AUTH" "${SEARCH_ENGINE_URL}/_cat/indices/${INDEXES}?h=index"

# Delete, one index at a time
for i in $(curl -s -u "$AUTH" "${SEARCH_ENGINE_URL}/_cat/indices/${INDEXES}?h=index"); do
  curl -s -u "$AUTH" -X DELETE "${SEARCH_ENGINE_URL}/${i}"; echo
done

Then delete the index template and the lifecycle policy, using the commands for your search engine.

OpenSearch (Amazon OpenSearch Service, or OpenSearch on RKE2 and OpenShift):

# Index template: the API depends on the OpenSearch version, so try both
curl -s -u "$AUTH" -X DELETE "${SEARCH_ENGINE_URL}/_index_template/${RELEASE}-log-explorer-index-template"; echo
curl -s -u "$AUTH" -X DELETE "${SEARCH_ENGINE_URL}/_template/${RELEASE}-log-explorer-index-template"; echo

# ISM policy
curl -s -u "$AUTH" -X DELETE "${SEARCH_ENGINE_URL}/_plugins/_ism/policies/${RELEASE}-log-explorer-index-policy"; echo

# Check that nothing is left
curl -s -u "$AUTH" "${SEARCH_ENGINE_URL}/_cat/indices/${RELEASE}-*?v"
curl -s -u "$AUTH" "${SEARCH_ENGINE_URL}/_plugins/_ism/policies/${RELEASE}-log-explorer-index-policy"; echo

exit

Elasticsearch (Elastic Cloud on Azure):

# Index template
curl -s -u "$AUTH" -X DELETE "${SEARCH_ENGINE_URL}/_index_template/${RELEASE}-log-explorer-index-template"; echo

# ILM policy
curl -s -u "$AUTH" -X DELETE "${SEARCH_ENGINE_URL}/_ilm/policy/${RELEASE}-log-explorer-index-policy"; echo

# Check that nothing is left
curl -s -u "$AUTH" "${SEARCH_ENGINE_URL}/_cat/indices/${RELEASE}-*?v"
curl -s -u "$AUTH" "${SEARCH_ENGINE_URL}/_ilm/policy/${RELEASE}-log-explorer-index-policy"; echo

exit

kubectl create namespace "$NAMESPACE"

jq -s '{apiVersion: "v1", kind: "List", items: .}' sdp-input-secrets.json | kubectl apply -n "$NAMESPACE" -f -

Otherwise, create them again exactly as described in the Configure Kubernetes page for your platform:

• Amazon Web Services
• Microsoft Azure
• Rancher RKE2
• Red Hat OpenShift

While recreating them, check the values that most often cause a first installation to fail:

• the SASL mechanism and login module match your messaging service: PLAIN for Azure Event Hubs, SCRAM for Kafka and Amazon MSK;
• the PostgreSQL TLS settings (sslmode in the JDBC URLs and in values.yaml) match what your database server requires. Managed services such as Amazon RDS and Azure Database for PostgreSQL usually require TLS;
• every key listed in the documentation is present: the prechecks in step 9 report missing keys.

helm template "$RELEASE" -n "$NAMESPACE" \
  oci://registry.na.semarchy.net/semarchy-release/semarchy-data-platform --version <CHART_VERSION> \
  --set global.prechecks.enabledPreflights=true \
  -f values.yaml | kubectl preflight -

Fix every Fail before installing. The preflight plugin is installed with Krew; see Run deployment prechecks.

Install without --wait or --atomic, with a longer timeout

helm upgrade --install "$RELEASE" \
  -n "$NAMESPACE" \
  oci://registry.na.semarchy.net/semarchy-release/semarchy-data-platform \
  --version <CHART_VERSION> \
  --timeout 20m \
  -f values.yaml

• Do not use --wait. Keycloak, billing and Data Management mount Secrets and ConfigMaps that post-install Jobs create. Helm with --wait waits for these pods before it runs the post-install Jobs, so it can only time out. The chart checks readiness itself with its rollout-check Job.
• Do not use --atomic. It rolls back an installation that was progressing normally, and leaves behind the external objects the Jobs had already created: exactly the situation this article cleans up.
• Use --timeout 20m, or more on slow networks. Helm's default of 5 minutes is often too short for the first image pulls.
• Keep the same release name, namespace, global.site_id and global.site_name as in your preparation. They are part of resource and state names.

While the installation runs, pods in CreateContainerConfigError or ContainerCreating are expected. They wait for a Secret or ConfigMap that a later Job creates. See the SDP Helm Chart description article to know which Job each pod waits for.

10. Troubleshooting tools

If the installation fails again, find the root cause before cleaning up again.

1. Find the first failing Job or pod. Later failures are usually consequences of an earlier one.

kubectl get jobs -n "$NAMESPACE"
kubectl get pods -n "$NAMESPACE"
kubectl logs job/<JOB_NAME> -n "$NAMESPACE"
kubectl describe pod <POD_NAME> -n "$NAMESPACE"

export CJ=$(kubectl get cronjob -n "$NAMESPACE" -l k8s.semarchy.net/diagnostic-install-job=true -o jsonpath='{.items[0].metadata.name}')
kubectl create job -n "$NAMESPACE" --from=cronjob/$CJ semarchy-diagnostic-install-job
export JOB=$(kubectl get job -n "$NAMESPACE" -l k8s.semarchy.net/diagnostic-install-job=true -o jsonpath='{.items[0].metadata.name}')
kubectl wait --for=condition=complete job/$JOB -n "$NAMESPACE" --timeout 10m
export POD=$(kubectl get pod -n "$NAMESPACE" -l k8s.semarchy.net/diagnostic-sidecar=true -o jsonpath='{.items[0].metadata.name}')
export LATEST=$(kubectl exec -n "$NAMESPACE" "$POD" -- sh -c 'ls -1t /diagnostic-volume/support-bundle*.tar.gz 2>/dev/null | head -n1')
kubectl cp -n "$NAMESPACE" "$POD:$LATEST" ./support-bundle.tar.gz