Collect a troubleshooting bundle
Collect a troubleshooting bundle from an Infrahub deployment and share it with OpsMill support. Collection is read-only and safe to run at any time, including while the instance is degraded. See Infrahub Collect for what a bundle contains and how secrets are masked. The same command produces the same bundle on Docker Compose and Kubernetes.
Prerequisites
Before collecting a bundle:
- Docker Compose
- Kubernetes
- The
infrahub-collectbinary is installed (installation guide) - Docker and Docker Compose are available to your user
- The Infrahub Compose project is running (fully or partially)
- The
infrahub-collectbinary is installed (installation guide) kubectlis configured with access to the cluster running Infrahub- Your role can read pod logs and execute commands in pods (
pods/logandpods/exec); no write or scale permissions are needed - To include traces from the deployment's Tempo (
--include-traceswithout--traces-tempo-url), your role can also find the Tempo service and forward a port to its pod:listandgetonservices,getandlistonpods, andcreateonpods/portforward
The binary is self-contained and collection works offline by default: it uses your existing Docker or kubectl access and performs no network access beyond your deployment itself. The one exception is the opt-in --benchmark, which downloads a benchmark image.
Step 1: Verify environment detection
Confirm the tool can find your Infrahub deployment:
infrahub-collect environment detect
The tool auto-detects Docker Compose projects and Kubernetes namespaces containing Infrahub. To list every deployment it detects:
infrahub-collect environment list
Step 2: Create the bundle
- Docker Compose
- Kubernetes
For a single Infrahub project, run:
infrahub-collect create
If several Infrahub projects run on the same host, target one explicitly:
infrahub-collect create --project=infrahub-production
If a single namespace contains Infrahub, run:
infrahub-collect create
To target a specific namespace:
infrahub-collect create --k8s-namespace=infrahub-prod
On Kubernetes, the bundle includes logs from every replica of each service, and for pods that crashed and restarted it also captures the previous container's logs — often exactly where the error is.
By default the bundle is written to ./infrahub_bundles. Use --output-dir to choose another location:
infrahub-collect create --output-dir=/tmp/support
Step 3: Monitor collection progress
The command streams progress as each collector runs:
INFO[0000] Detecting deployment environment...
INFO[0000] Kubernetes environment detected (namespace: infrahub-prod)
INFO[0000] Creating collection ID: 20260702_141530
INFO[0001] Collecting logs: infrahub-server (2 replicas)
INFO[0004] Collecting logs: task-worker (3 replicas, 1 restarted - including previous logs)
INFO[0009] Collecting database logs...
INFO[0012] Collecting message-queue status...
INFO[0013] Collecting cache status...
INFO[0014] Collecting task-manager state...
INFO[0016] Collecting server info...
INFO[0017] Collecting container metrics...
INFO[0018] Creating compressed archive...
INFO[0021] Bundle created: infrahub_bundles/support_bundle_20260702_141530.tar.gz
If a service is down or unreachable, collection continues with the remaining services and records the failure:
WARN[0013] Collector failed: cache status (container not running) - continuing
A partial bundle is still a useful bundle — a degraded instance is precisely when you need one.
Step 4: Verify the bundle
Check the archive and review what was collected:
# Verify archive integrity
tar -tzf infrahub_bundles/support_bundle_20260702_141530.tar.gz > /dev/null && echo "Archive is valid"
# View the collection manifest
tar -xzOf infrahub_bundles/support_bundle_20260702_141530.tar.gz bundle/bundle_information.json | jq '.'
The manifest records what was attempted and what succeeded:
{
"manifest_version": 2026110500,
"collect_id": "20260702_141530",
"created_at": "2026-07-02T14:15:30Z",
"tool_version": "1.2.0",
"infrahub_version": "1.5.2",
"environment": "kubernetes",
"edition": "enterprise",
"helm": {
"release_name": "infrahub",
"chart": "infrahub-enterprise",
"chart_version": "1.2.3"
},
"log_lines": 100000,
"collectors": [
{"name": "logs/infrahub-server", "status": "success"},
{"name": "logs/task-worker", "status": "success"},
{"name": "cache-status", "status": "failed", "reason": "container not running"},
{"name": "benchmark", "status": "skipped", "reason": "not requested"}
]
}
The edition field reports whether the deployment runs Infrahub Community or Enterprise. On Docker Compose it is classified from the infrahub-server container image, and on Kubernetes from the Helm chart name. It is omitted when the edition cannot be determined.
The bundle layout is identical on Docker Compose and Kubernetes. Inside the archive, all files live under a top-level bundle/ directory:
bundle_information.json- Collection manifestlogs/<service>/- Container logs, one file per replica (plus*.previous.logfor restarted pods)database/- Neo4j server logsmessage-queue/- RabbitMQ queues, exchanges, connections, and statuscache/- Redis info, clients, configuration, and slow logtask-worker/- Task worker state, one directory per replicatask-manager/- Work pools, work queues, recent flow runs, the pending and running flow and task runs, events, and automationsserver/- Version, installed packages, API information, configuration, schema, worker schema-sync status (infrahub_status.json), environment variablesmetrics/- Container resource metricstraces/- Traces of slow requests, one OTLP JSON file per trace, slowest first (only with--include-traces)
Each file holds the output of the command that produced it and nothing else, so a .json file always parses. When a command fails, its file is absent and a sibling <name>.err.txt records what ran, how it failed, and what the command reported — for example task-manager/flow-runs.err.txt in place of flow-runs.json. The manifest lists the same failure as a reason on the collector.
Masking is based on key names: environment variables and configuration keys containing password, secret, token, or key are replaced with ******** before they are written to the bundle. Secrets stored under other key names are not detected. Service logs are collected as-is, and database query logs (--include-queries) can contain customer data. Review the bundle contents before sharing.
Step 5: Share the bundle with support
Send the archive to OpsMill support through your usual support channel. The bundle is a plain local file, so environments with restricted egress can transfer it through whatever mechanism their security policy allows.
Advanced usage
Include a backup for issue reproduction
Support often asks for a backup alongside logs so they can reproduce your issue locally. Create both in one run:
infrahub-collect create --include-backup
This runs the standard infrahub-backup logic and records the backup in the bundle manifest.
Unlike log collection, the backup step follows the normal backup behavior for your deployment, which can include stopping application containers while the snapshot is taken. Plan for a short interruption, or skip --include-backup and create the backup separately during a maintenance window.
Include database query logs
Query logs can help diagnose slow or failing database operations, but they may contain customer data, so they are excluded by default:
infrahub-collect create --include-queries
Adjust the log volume limit
By default, collection captures the most recent 100,000 log lines per container. Raise or lower the limit as needed:
# Collect more history for a long-running issue
infrahub-collect create --log-lines=500000
# Or via environment variable
export INFRAHUB_LOG_LINES=500000
infrahub-collect create
The effective limit is recorded in the manifest, so support knows whether logs were truncated.
Include the traces of slow requests
When Infrahub is slow, the traces of the slow requests show which component and which database query took the time. If your deployment sends traces to Tempo, for example with the Infrahub observability stack, add them to the bundle:
infrahub-collect create --include-traces
By default, the collector reads the traces of the last 2 hours that lasted longer than 1 second. Change the window and the minimum duration as needed:
# A slowness that is still going on: the last 30 minutes
infrahub-collect create --include-traces --traces-since=30m
# A slowness that ended: an absolute window, in RFC 3339
infrahub-collect create --include-traces \
--traces-start=2026-11-04T14:00:00Z --traces-end=2026-11-04T15:00:00Z
# Only the traces longer than 5 seconds
infrahub-collect create --include-traces --traces-min-duration=5s
The collector writes one file per trace to traces/, numbered from 0001 for the slowest trace. It stops when the files reach the size limit, 100 MB by default (--traces-max-size, in megabytes of uncompressed JSON), or after 5 minutes of fetching. Because the slowest traces are written first, the traces left out are always faster than the traces kept. The traces entry of bundle_information.json records the window, the minimum duration, and how many traces were found, written and left out.
Personal data is removed. The collector keeps only these fields of each trace:
- On every span: the trace, span and parent span IDs, the span name and kind, the start and end times, and the status code.
- On the process that sent the span:
service.name,service.versionandworker.id. - On database query spans (
execute_db_query_with_metadata):query_nameandrows. - On GraphQL spans (
execute_graphql):type,operation,nameandbranch. - On HTTP server spans:
http.method,http.routeandhttp.status_code.
Everything else is removed before the file is written, including the text of database queries, URLs, client IP addresses, user agents, exception messages and span events.
Where the collector finds Tempo. The collector sends its requests from the machine where it runs, so it works while the Infrahub containers or pods are down:
- On Docker Compose, it reads the published port 3200 of the
infrahub-temposervice of the project. With a remote Docker context, this port is on the remote host; use--traces-tempo-urlinstead. - On Kubernetes, it finds the service labelled
app.kubernetes.io/name=tempoin the Infrahub namespace and forwards a local port to its port 3100 for the duration of the collection. Theinfrahub-observabilityHelm chart creates this service.
The collector works with Tempo 2.5 and later.
When no Tempo is found, when Tempo does not answer the search within 60 seconds, or when your Kubernetes role lacks a permission, the traces entry is skipped and the reason names what is missing.
Your own trace backend. To read the traces from another Tempo-compatible backend, give its URL. If the backend needs a token, set it in your shell; the collector sends it as a bearer token and never writes it to the bundle or the output:
export INFRAHUB_TRACES_TEMPO_TOKEN=<token>
infrahub-collect create --include-traces --traces-tempo-url=https://tempo.example.com
When the variable is not set and the command runs in a terminal, the collector asks for the token without showing what you type. An empty answer means no token.
The trace sampling ratio. Infrahub records a trace for only a share of the requests. To let support know that share, give an Infrahub API token of a user with the super_admin permission. The collector uses it only to read the sampling ratio (Infrahub 1.12 or later) and records the ratio in the bundle:
export INFRAHUB_TRACES_API_TOKEN=<token>
infrahub-collect create --include-traces
Without the variable, the collector asks for the token in a terminal; leave the answer empty to skip. If the slowness can be reproduced, a super admin can raise the ratio for a limited time with PUT /api/troubleshooting/trace-sample-ratio before reproducing it, and remove the temporary ratio with DELETE on the same endpoint after the bundle is created.
Tempo can take about a minute before a new trace appears in its search. If you create the bundle right after the slowness, wait a minute first.
Run a performance benchmark
For performance investigations, support may ask you to include a benchmark run:
infrahub-collect create --benchmark
The benchmark requires downloading the OpsMill benchmark container image and measures whether the host meets Infrahub's resource requirements — disk IOPS, memory, and single-core CPU — rather than generating load against your instance. If the image cannot be pulled — for example in an air-gapped environment — the benchmark is skipped with a warning and the rest of the collection completes normally.
Troubleshoot collection
No deployment detected
If infrahub-collect create reports that no environment was found, list what the tool can see:
infrahub-collect environment list
On Docker Compose, confirm the project is running with docker compose ls; the project name must contain infrahub, or you must name it with --project. On Kubernetes, the namespace is discovered through the app.kubernetes.io/name=infrahub label; for deployments with custom labels, name the namespace with --k8s-namespace.
Permission denied on Kubernetes
Log collection needs the pods/log permission and the status collectors need pods/exec. If your role lacks one of them, the affected collectors are recorded as failed in the manifest. Ask your cluster administrator for read access to both — collection needs no write or scale permissions.
With --include-traces, the collector finds the deployment's Tempo and forwards a port to it, which needs list and get on services, get and list on pods, and create on pods/portforward. kubectl port-forward reads the service and the pod behind it before it opens the port. Without these permissions, the traces entry is skipped and the reason names what is missing. With --traces-tempo-url, the collector reads a URL that your machine reaches and needs none of them.
Docker or kubectl not found
The tool depends on the Docker CLI or kubectl being installed and configured on the machine where it runs. When neither is usable, the command fails immediately and the error names the missing CLI.
Individual collectors failed
Collector failures on a degraded deployment are expected — a stopped cache container cannot answer status queries. The command still exits successfully, and the manifest records each failure with a reason. Send the partial bundle as-is; support reads the manifest first.
Validation
Confirm your collection works:
-
infrahub-collect environment detectfinds your deployment - Bundle archive is created and passes the integrity check
- Manifest lists an explicit outcome for every collector
- Logs are present for every service replica
- Masked environment output contains no plaintext secrets