Tenant management and sandboxes

A tenant has its own database, its own Elasticsearch indexes (under the tenant's index prefix) and its own DMS files. Tenants share the installed backend and frontend version. The Camunda process history indexes are shared by all tenants unless history isolation is enabled (see Limitations).

The same mechanism supports:

• Feature development on DEV — each developer or feature gets a sandbox copied from the DEV configuration, changes it without affecting others and exports the result to Git.
• E2E, regression and load tests — a short-lived tenant per test run, isolated from the data of other users.
• Project work on a customer configuration — copy the current customer configuration into a new tenant, develop and test there, then deliver the reviewed configuration package to the customer environment.
• Organizations — a long-lived tenant for another organization on the same installation.

Creating a sandbox does not deploy a separate application build. Changes to Java/Kotlin/Angular code follow the usual application build and deployment process; a tenant isolates configuration and data.

Configuration

Tenant management is disabled by default. Put the following into the standard tSM configuration (Spring Cloud Config) shared by all backend services and the gateway:

tsm:
  redis:
    cache:
      enabled: true
  multitenant:
    enabled: true
    management:
      enabled: true
      urls:
        - https://sandbox01.example.test
        - https://sandbox02.example.test
      wildcard-domain: tsm-dev.example.test   # optional, replaces the pool for new tenants
      wildcard-separator: "-"                 # "." (default) or "-"
      cleanup-grace: 3d
      poll-interval-ms: 10000
      cleanup-max-attempts: 16
      cleanup-retry-backoff: 30s

• tsm.multitenant.enabled turns on tenant routing of database connections (per-tenant data sources and Flyway migrations of tenant databases).
• tsm.multitenant.management.enabled turns on managed provisioning, the URL pool and gateway tenant routing. It must be set in the configuration shared by all backends, not only in user management and the gateway. Every backend that creates Elasticsearch indexes evaluates the flag locally. A backend without it creates indexes without the tenant allocation marker (_meta.tsmTenantAllocation), for example during an admin reindex in a sandbox, and deleting that sandbox later fails on the missing marker.
• tsm.redis.cache.enabled enables the installation's shared Redis client (TSM_REDIS_CONFIG / redisson.yaml), which management uses for allocation locks.
• tsm.multitenant.management.urls is the prepared URL pool. Each entry must be an HTTPS origin on port 443 without a path, and hosts must be unique. Prepare DNS, ingress and TLS for every pool host so that they all reach the common gateway. The gateway must receive the original Host; the proxy must not rewrite it to an internal service name. An unassigned pool URL returns 404.
• tsm.multitenant.management.wildcard-domain (empty by default, which keeps the pool) lets each new tenant derive its URL from the tenant code instead of taking one from the pool: https://<code><separator><domain>. The pool urls then only route tenants that already hold such a URL. A host below the domain that no tenant holds returns 404.
• tsm.multitenant.management.wildcard-separator joins the code and the domain. With . (default) a tenant gets sandbox01.tsm-dev.example.test, which needs DNS and a certificate for *.tsm-dev.example.test. With - it gets sandbox01-tsm-dev.example.test, a single label that the DNS wildcard and the certificate of the parent domain (*.example.test) already cover. With - the tenant code (up to 40 characters), the dash and the first label of the domain must fit one 63-character DNS label; otherwise the configuration is rejected at startup.

The ingress API does not accept a partial wildcard such as *-tsm-dev.example.test as a host. For the - form, create one anchor host (for example tenants-tsm-dev.example.test) and put the pattern into the ingress-nginx server-alias annotation, for example ~^[a-z0-9-]+-tsm-dev\.example\.test$. A host that has its own Ingress takes precedence over the pattern. Add nginx.ingress.kubernetes.io/force-ssl-redirect: "true", because hosts matched through the alias are otherwise not redirected from HTTP to HTTPS, and repeat the same annotations on any separate Ingress that carries the AI stream paths for these hosts. Browser calls from a tenant host to its own gateway are same-origin, so they need no CORS entry.

All management properties and their defaults are listed in the Configuration Properties Reference.

With management enabled, the legacy tenant registry CRUD/export endpoints and the synchronous provisioning are disabled and replaced by the managed API with asynchronous Jobs. Migrate their consumers before switching it on.

Installations that already have non-default tenants with FILESYSTEM DMS storage must move files before enabling management; see Limitations.

Frontend

tsm-ui 2.4 has no dedicated tenant management screen. Administrators manage tenants through the REST API, directly or with the tsm CLI (see Managing tenants with the API and CLI).

The login page shows the tenant field only when the UI parameter showTenant allows it (it does when the parameter is not set; "showTenant": false in the UI params.json hides it) and the public endpoint GET userManagement/open/tenants returns at least one tenant. That endpoint exists only on installations without tenant management, so with tsm.multitenant.management.enabled=true the field is never shown and showTenant has no effect. A managed tenant is entered through its assigned URL (the gateway selects the tenant by the URL) or, in the API, with the X-Tenant-Id header.

Signing in with Google

The Google button on the login page works only on addresses registered in the Google OAuth client. The address of a tenant derived from wildcard-domain and pool addresses are not registered there unless an administrator adds them (see below). Google then shows an error (redirect_uri_mismatch) and the user is not signed in. On such a tenant sign in with the user name and password; for API calls use an API key.

Administrators can allow Google on the address of a long-lived tenant: add <address>/account/redirect as a redirect URI and the address as an authorized origin in the Google Cloud Console OAuth client. Do not do this for sandboxes that are created and dropped regularly.

To hide the button on addresses where Google would fail, list the registered addresses in the UI params.json: "googleSignInOrigins": ["https://tsm-dev.example.test"]. The button is then shown only on those addresses. "showGoogleSignIn": false hides it for the whole deployment.

Permissions

Standard privileges Um.Tenant.View, Um.Tenant.Add, Um.Tenant.Edit and Um.Tenant.Delete control tenant administration. Administration is available only in the default tenant; it is denied in every other tenant, including for users with Um.Tenant privileges. The backend checks this independently of the client. A copied administrator in a target tenant can use that tenant but cannot manage tenants.

Managing tenants with the API and CLI

The tenant management API is tsm-user-management/api/v2/tenant-management. Call it in the default tenant. With the tsm CLI the global --tenant option (or the TSM_TENANT environment variable) sets the X-Tenant-Id header:

tsm --tenant default api GET tsm-user-management/api/v2/tenant-management/sources
tsm --tenant default api GET tsm-user-management/api/v2/tenant-management/profiles
tsm --tenant default api POST tsm-user-management/api/v2/tenant-management --input tenant-create.json
tsm --tenant default api GET tsm-user-management/api/v2/tenant-management

Example tenant-create.json:

{
  "code": "sandbox01",
  "name": "Sandbox for feature X",
  "sourceTenant": "default",
  "purpose": "SANDBOX",
  "ttlDays": 7
}

Field	Description
code	One lowercase letter followed by at most 39 lowercase letters or digits. default and admin are reserved. Codes are never reused.
name	Display name.
sourceTenant	Required. A code returned by /sources; it is never inferred from the caller. The source must be READY and not expired.
purpose	ORGANIZATION, SANDBOX (default), E2E, REGRESSION or LOAD_TEST.
ttlDays / unlimited	Lifetime in days (1–3650) or unlimited; not both. Defaults: sandbox 7 days, E2E/regression/load test 1 day, organization unlimited.
creationProfile	Creation profile from /profiles. Required for a remote source.

Other operations ({code} is the tenant code):

Request	Privilege	Purpose
POST /{code}/lifetime	Um.Tenant.Edit	Change the lifetime (ttlDays or unlimited).
POST /{code}/cancel	Um.Tenant.Delete	Cancel a REQUESTED, PROVISIONING or INITIALIZING allocation.
POST /{code}/delete	Um.Tenant.Delete	Request deletion.
POST /{code}/confirm-quiescent	Um.Tenant.Delete	Confirm that no writer is still writing into the target resources (see below).

The list returns the status, the assigned URL, the provisioning Job, timings and the error of a failed allocation.

Working in a tenant

Once the tenant is READY, open its assigned URL, or select it with the X-Tenant-Id header (--tenant in the CLI) on the base URL. When both are present, the URL and the header must agree, and the tenant in a signed token must agree with the resolved tenant. The URL or header alone grants no access.

Configuration packages are exported and installed per tenant with the usual tsm CLI commands and the --tenant option:

tsm --tenant sandbox01 package download AllConfig
tsm --tenant sandbox01 package install AllConfig

A typical development flow:

1. Create a sandbox from the source tenant (for example DEV default).
2. Change forms, processes, scripts, registers and other configuration in the sandbox. Prepare business test data separately; business data is not copied.
3. Export the configuration with the tsm CLI into a feature branch, review the Git diff and create an MR. ChangeSets are optional.
4. After approval, install the reviewed configuration package into the destination tenant. The sandbox database is never cloned into the destination.
5. Extend the lifetime, or delete the sandbox when it is no longer needed.

Lifecycle

Status	Meaning
REQUESTED	Allocation accepted, waiting for the provisioning Job.
PROVISIONING	Database and Elasticsearch indexes are being prepared.
INITIALIZING	Data is transferred; initializers (index rebuild, data packages) run. The tenant accepts ordinary routing and authentication, but is not offered as a source or for E2E setup.
READY	Available.
EXPIRING	Lifetime expired; still available during the grace period before cleanup.
DELETING	Cleanup requested or running; the tenant is not available.
DELETE_FAILED	Cleanup gave up after the configured number of attempts.
DELETED	Resources removed. The code stays reserved.
FAILED	Provisioning or initialization failed. The allocation stays visible for diagnosis.

After expiresAt the tenant becomes EXPIRING. The cleanup deadline deleteAfter is stored when the lifetime is set (expiry plus tsm.multitenant.management.cleanup-grace, 3 days by default); changing the grace later does not move existing deadlines. After deleteAfter the tenant is unavailable even if cleanup keeps failing. A source tenant that still has copies stays EXPIRING and available until the copies are gone, and it cannot be deleted while dependent tenants exist.

A failed cleanup in DELETING is retried with exponential backoff: the first retry after cleanup-retry-backoff (30 s), each next one after double the delay, at most one hour. After cleanup-max-attempts failed passes (16, roughly 9 hours with the defaults) the tenant ends in DELETE_FAILED and the job no longer touches it. DELETE_FAILED keeps its URL and code.

Cancelling and recovering

• Cancel records the intent for a REQUESTED, PROVISIONING or INITIALIZING allocation. Provisioning stops between stages and cleanup starts after all copy writers finish. The final stage (the FINALIZING preparation phase) cannot be cancelled.
• FAILED or DELETE_FAILED — find the cause in the service log, fix it and call POST /{code}/delete again. The tenant returns to DELETING with a fresh retry budget. Failed allocations are not resumed automatically; create a new tenant instead.
• Uncertain writers — when a copy or initializer ended uncertainly (for example the pod stopped during an Elasticsearch copy), the list shows quiescentConfirmationRequired=true and deletion is rejected. Verify that no writer still runs (for a same-cluster copy, check the reindex tasks in Elasticsearch _tasks; for an initializer, check the logs of the target services), then call POST /{code}/confirm-quiescent and request the deletion.

E2E tests in a tenant

The E2E runner works per tenant:

tsm --tenant e2e01 e2e initialize --yes
tsm --tenant e2e01 e2e run --suite <suite>

• tsm e2e initialize creates the optional E2E configuration storage in the selected tenant and requires an explicit --tenant and --yes.
• tsm e2e run requires --tenant or TSM_TENANT.
• Integration endpoints and credentials for a testing tenant (any purpose other than ORGANIZATION) are set per tenant with GET/PUT/DELETE tsm-user-management/api/v2/tenant-management/{code}/testing-configurations[/{prefix}]. Overrides require a READY managed testing tenant.
• In the managed modes the runner clones the source tenant, waits for READY and logs in to the clone as the test actor. The actor must already be an active member of the source tenant; the runner does not add the actor or assign roles in the clone, the user and roles arrive with the copy. In the EXISTING mode the runner creates no tenant.

Copying a customer configuration

By default a tenant is copied from a local source registered in the same installation. To start from another installation (typically the current configuration of a customer), the deployment defines remote sources and creation profiles in a private JSON file referenced by tsm.multitenant.management.preparation-profiles-file:

{
  "profiles": {
    "core": {
      "version": "1", "copyTables": {"am": ["amt_region", "amt_layer"]},
      "emptyTables": {}, "validationTables": {"am": ["amt_region"]},
      "copyAliases": ["registervalue", "catalogtree", "relationshiptree", "ticket-template", "configtypecontent"],
      "requiredAliases": ["registervalue", "catalogtree", "configtypecontent"]
    }
  },
  "creationProfiles": {
    "fast": {"name": "Core baseline", "copyProfile": "core", "allowedSources": ["default"], "elasticMode": "COPY"},
    "rebuild": {
      "name": "Core with rebuilt indexes", "copyProfile": "core", "allowedSources": ["default"],
      "elasticMode": "REBUILD", "initializers": ["native-elasticsearch-rebuild-v1"]
    }
  },
  "remoteSources": {}
}

• profiles are database-copy profiles: which tables are copied, which stay empty and which Elasticsearch aliases are copied. Without copyAliases the profile copies registervalue, catalogtree, relationshiptree, ticket-template and configtypecontent. Copy speed is tuned with elasticBatchSize, elasticParallelism, elasticNativeBatchSize and elasticMaxBulkBytes.
• creationProfiles choose a copy profile, the allowed sources and the Elasticsearch mode:
  • COPY copies the listed indexes from the source;
  • REBUILD creates empty indexes and rebuilds them in the target. The profile must list the native-elasticsearch-rebuild-v1 initializer in initializers, otherwise provisioning fails with "REBUILD requires an installed native target rebuild initializer". The rebuild is bounded by native-rebuild-timeout (30 minutes by default);
  • NONE creates empty indexes.
• initializers lists the initializers that run in the new tenant. requiredDataModelStatuses lists the Data Model readiness statuses (CURRENT, PARTIAL, FAILED, STALE, DELETED) that the new tenant is verified against before it becomes READY; it cannot be combined with NONE.
• A creation profile can also import data packages (dataPackages, each with path, sha256, packageId, version and records, the expected record count). The profile must list data-package-v1 in initializers; without it the tenant is rejected at creation with "Data packages require their initializer". The tenant becomes READY only after the import is verified.
• remoteSources bind a source name to its database and Elasticsearch connection ({name, database: {host, port, dbName, username, password}, elasticsearch: {url, authorization?, compatibleVersion}, elasticsearchPrefix, excludedElasticsearchPrefixes}). Render them from secrets during deployment. Remote source IDs must not collide with local tenant codes. The API exposes only names and profile codes, never the bindings. A remote source requires an explicit creation profile.

Copy scope

The default profile copies configuration, registers, users (including password hashes), roles and privileges from the selected sourceTenant. Every tenant has its own users. Copying provisions a fresh owned target database and recreates the supported schema; excluded runtime tables stay empty. This is not a full database clone. Business data, running processes, history, regular DMS content, sessions, refresh tokens and API keys are not copied. Runtime credentials are not copied; configure them separately for the target. Business Elasticsearch indexes start empty; the configuration indexes registervalue, catalogtree, relationshiptree, ticket-template and configtypecontent are copied from the source.

Limitations

• The source database and the tenant registry must currently be on the same PostgreSQL server.
• The copy is not an atomic snapshot of the source. For a reproducible baseline keep the source configuration unchanged during copying, or align the result afterwards with a versioned export/import.
• DMS FILESYSTEM storage of existing tenants. With management enabled, DMS stores files of non-default tenants under <storage-root>/tenants/<tenant>/<bucket>/<attachment-id> (the default tenant keeps its original path). Files of non-default tenants created before management was enabled are not moved automatically: downloads then fail with 404 (the content endpoint treats empty content as not found; only expand=Content returns empty data), previews are missing and deletes do not find the file. Move them manually before enabling management.
• Camunda history. The Camunda history indexes are shared by all tenants. Per-tenant filtering requires camunda.history.elasticsearch.tsm-tenant-isolation.enabled=true in every service with tsm-camunda. Deploy the new tsm-camunda to all those services before enabling the flag; a service with an older version writes history without the owner. The application does not migrate existing documents: they have no owner and are hidden after enabling the flag until they are updated again. Assign them beforehand, for example with an _update_by_query that sets tsmTenantId=default while no other tenant has written history yet. See history-tenant-isolation.md.
• Background processing. Scheduled jobs that run without a user token resolve to the default tenant. Kafka listeners run in the tenant named in the tenant header of the record; only records without that header are processed in default. Processing Camunda jobs (timers, asynchronous continuations) across tenants is not supported yet.
• Upgrades. On service start Flyway migrates the default database and every READY or EXPIRING tenant database. Check the startup logs of each service after an upgrade.
• Non-default tenants depend on user management at runtime: their database connections are resolved through user management, so an unavailable user management blocks them (the default tenant is not affected).

Implementation reference

The technical guide lives with the implementation: Správa tenantů — TSM-2567. It describes routing and token validation, shared service identities, Redis and Job coordination, copy details, DMS paths and verification commands. Customer DNS/proxy/ingress implementations may differ; provisioning is implemented in tSM services.