Troubleshooting, updates and backups

When something goes wrong, follow this page from top to bottom. First you find out what the problem is (error code, system events, logs), then the fix in the common problems tables. If that does not solve it, the end of the page says what to send to support.

#The first 5 steps when something is wrong

1. If the screen shows an Error code, copy it (see below). It is the most valuable detail in any conversation with support.
2. If you are an installation admin, open the System events page and read the warnings and errors from the same time.
3. If the problem is about a runner, check its state on the Runners page.
4. Look for the symptom in the common problems tables below.
5. If it is still not solved, take a bundle with Download support bundle and e-mail it to support.

#Error codes and request ids

When a server error (5xx) happens in the UI, the message shows an Error code with a copy button. The code is the request's X-Request-ID, and it also appears:

• on every line of the controller log for that request, as request_id,
• in the Request id column of its event on the System events page.

So one code lets you follow what happened to that request end to end.

Step by step:

1. Press the copy button next to the Error code in the error message.
2. On the System events page paste it into the search box (message or request id) and press Filter.
3. Open the event's row: the message, the source and, if there is one, the Stack trace show.
4. To search the log: grep <error code> controller.log (where the log is: below).
5. Always include this code when you write to support.

#System events

System events (installation admins, Infrastructure → System events in the left menu) lists what the controller noticed: panics (recovered, with stack traces), server errors (5xx), runners connecting and dropping, failed runs, license, the update check, the database, SSO/LDAP and notification channel errors. Events are kept for 30 days, at most 20,000.

System events page

The top of the page:

• Panics recovered: N — the panics the controller caught in its own code. If it is above zero, always send the support bundle.
• "The controller log is also written to the data volume (rotated, 20 MiB × 5)." or "The controller log goes to stdout only (no SPITFIRE_LOG_FILE): it is lost when the container is re-created." — whether the log survives.
• "N events could not be stored because the database fell behind." — shows when the database is overloaded.

Filtering, step by step:

1. Narrow From and To to the hours of the problem.
2. Level: All, Warnings and errors or Errors only.
3. Source: Panic, API (5xx), Runner, Run, License, Update check, Database, SSO / LDAP, Notification channels, Integrations, Controller, Scheduler, Synthetic monitoring, Mobile push, or All sources.
4. Search: part of a message or a request id.
5. Press Filter; Clear filters resets everything.

Each row shows Time, Level (info, warning, error), Source, Event and Request id. If the same event repeated within a minute, the count shows next to it. Details and the Stack trace show when you open the row.

At the bottom, Recent warnings and errors in memory shows the latest warning and error lines the controller logged since it started, newest first, with secrets removed. They reset when the controller restarts, so look at them before restarting.

#Support bundle

The support bundle gathers what is needed to understand the problem into one zip. Spitfire never sends this bundle anywhere on its own: you download it, look inside if you like, and e-mail it to support yourself.

#From the UI

1. On the System events page press Download support bundle at the top right.
2. The dialog shows each File and its Size (the zip is usually much smaller, being compressed).
3. Read what is left out.
4. Press Download.

Support bundle dialog

#When the UI does not open (command line)

If the controller keeps restarting, or nobody can sign in, in the install folder (~/spitfire by default):

./install.sh --support-bundle                 # Docker (the install in this folder)
./install.sh kubernetes --support-bundle      # Kubernetes (--namespace / --context)

On Windows (Docker Desktop), in PowerShell:

.\install.ps1 -SupportBundle

The installers collect the container logs (docker compose logs / kubectl logs, also the previous container's after a crash), the docker compose ps / kubectl get pods output and the settings with secrets removed, and hand them to the controller, which adds them to the same zip under cli/. If the controller cannot, the zip holds only what the script collected. The script prints where the file is; the file stays on that machine.

#What is in it

Left out: test definitions, request/response bodies, data files, connection details, user names and e-mail addresses, the license key and its signature. Any setting not on the allowlist of harmless settings becomes [redacted]. In the logs, secret values and anything that looks like a credential (a password in a URL, bearer and Spitfire tokens, password=, client_secret=, chat webhook URLs) become [redacted], and e-mail addresses [e-mail].

#Logs

#Where

The controller also writes its log to a file on the data volume, which survives re-creating the container (Docker: the logs volume; Kubernetes: an emptyDir that survives container restarts but goes with the pod):

To follow it live (Docker):

cd ~/spitfire
docker compose -p spitfire -f deploy/docker/docker-compose.yml logs -f --tail 200 controller

#How to read it

In the default text format each line is key=value pairs:

time=2026-10-08T09:41:12.318+03:00 level=ERROR msg="…" request_id=4f1c…

• level: INFO, WARN, ERROR. Look at ERROR lines first.

• msg: what happened.

• request_id: the same as the Error code in the UI; to find all of that request's lines:

  bashCopy

  docker compose -p spitfire -f deploy/docker/docker-compose.yml logs controller | grep 4f1c

For more detail set SPITFIRE_LOG_LEVEL=debug on the controller and restart it; set it back to info once the problem is solved.

#Common problems

#Runner offline

Runner states on the Runners page: Idle, Running, Draining, Offline.

Runners page

#License limits

#Connection and network errors

#Write protection refusals

A test with steps that change data (SQL/Mongo writes, a dangerous Redis command, an HTTP request marked as modifying data) needs explicit confirmation everywhere:

#Update problems

#Database

#Updating

When a new release is out, installation admins see a banner at the top ("Spitfire … is out"); What's new and how to update shows the command. The same is always on the License page, in the Version and updates card: Installed version, Installation, Update check, Outbound proxy and Update command.

Version and updates card

Step by step:

1. Do not update while a run is going. Wait for running runs to end.

2. Copy the command from the Version and updates card on the License page (or check again with Check now).

3. Run it on the server where the controller is installed:

   bashCopy

   # Linux / macOS (Docker)
   curl -fsSL https://spitfire.tr/install.sh | bash -s -- docker
   # Kubernetes
   curl -fsSL https://spitfire.tr/install.sh | bash -s -- kubernetes

   powershellCopy

   # Windows (PowerShell)
   irm https://spitfire.tr/install.ps1 | iex

   If you installed into another folder, the card gives the command with SPITFIRE_DIR=….

4. The command first backs up the database (when the version changes), keeps your settings (deploy/docker/.env or the spitfire-secrets Secret) and data, and installs the new release.

5. Runner servers in other locations have their own command on the card (curl -fsSL https://spitfire.tr/install.sh | bash -s -- runner). Keep remote runners on the controller's release; the Runners page marks a different one in yellow. For runners without Docker (systemd, Windows service), the Update button or Update runners automatically is enough.

#Backups and restore

#What is backed up automatically

• On every update that changes the version, the database is first dumped to ~/spitfire/backups/*.dump. The newest 5 are kept. --no-backup (Windows: -NoBackup) skips it; not recommended.
• Back up the keys yourself as well: the deploy/docker/.env file (Docker) or the spitfire-secrets Secret (Kubernetes). If the SPITFIRE_SECRET_KEY in it is lost, stored connection secrets cannot be read.

#Rolling back

If an update went wrong, in the install folder:

~/spitfire/install.sh docker --rollback          # Docker
~/spitfire/install.sh kubernetes --rollback      # Kubernetes
~/spitfire/install.sh docker --rollback ~/spitfire/backups/<dump>.dump   # a specific dump

.\install.ps1 -Rollback                          # Windows
.\install.ps1 -Rollback -RollbackDump <file>

What happens:

1. The current database is dumped first, too.
2. The chosen dump (the newest by default) is restored.
3. The release that took that dump is started.

Changes made after that dump are lost. Running it again rolls forward. Without a terminal (e.g. from a script) it needs --yes.

#Contacting support

If none of the above solved it, write to support:

• Turkish: iletisim@spitfire.tr
• English: contact@spitfire.tr

Include the following (a complete e-mail speeds the fix up by days):

1. What happened? What you expected and what you saw, short and concrete.
2. When? Date, time and time zone.
3. The error code (if any) — the Error code / request id from the UI.
4. The support bundle — System events → Download support bundle, or ./install.sh --support-bundle. You can look inside before sending it.
5. Version and install kind — Installed version and Installation (Docker, Kubernetes, Windows) from the License page.
6. Steps — what you did to reproduce it; a screenshot helps.
7. For a run, its link or id; for CI, the CLI output (with the token removed) and the exit code.

#Related pages

• CI/CD integration — exit codes and CI errors
• Schedules and monitoring — notification channel and schedule problems
• Observability — when backend data does not arrive
• Mock, chaos and capacity
• System events In Spitfire: /system-events · Runners In Spitfire: /runners · License In Spitfire: /license