Troubleshooting Your Application

View Docker Services

Whenever you deploy a container with Docker, they are grouped into units called "services". Services could be things like databases, web servers, queues, etc.

sudo docker service ls

Learn more about "docker service ls" →

View Docker Service Logs

When you list out your services, you can now view the logs of that service.

sudo docker service logs <service_name>

Learn more about "docker service logs" →

Running commands in a running container

Sometimes you might need to do some troubleshooting in a running container. To do this, we need to follow this process:

1. Determine the running container ID for the service.
2. Use docker exec (with a TTY and interactive mode) attach to the shell of the running container.

We will use docker ps to get the container ID, then use docker exec to open a shell to the container.

# Get the container ID of the running service
sudo docker ps --filter "name=<service_name>" --format "{{.ID}}"

# Run a command in the running container (you can replace "sh" with any command)
# but by default, this will open a shell to the container (type "exit" to close)
sudo docker exec -it <container_id> sh

This is a two step process, but if you want to do this in a single command, we can use this trick below:

# Replace <service-name> with the name of your service. This will run `sh` in the first container ID it finds.
# You can change `sh` to `bash` if your container. Bash will give you a better experience.
sudo docker exec -it $(sudo docker ps --filter "name=<service_name>" --format "{{.ID}}" | head -n 1) sh

We're able to run this in a single command because we're using a subshell to get the container ID and then passing that to the docker exec command.

If you don't need a shell and you want to run something like php artisan migrate, you would just replace sh with php artisan migrate. Just make sure your container is running commands relative to the directory you want to run the command in. Sometimes it's best to just open a shell and navigate to the directory you want to run the command in.

Viewing application logs

When you're viewing application logs, you might have multiple places to look depending on the configuration of your application and what output you're trying to look for.

1. Docker Swarm Service Logs - The output here will be more server centric. Meaning you'll see deeper services logs such as PHP, FPM, Nginx, etc.
2. Application Logs - The output here will be more application centric. Meaning you'll see logs from your application itself.
3. Error Tracking Services - If you're using a logging service like Sentry or GlitchTip (which we highly recommend), you'll want to check those logs as well. These services will also alert you if there are any critical errors happening in your application.

Viewing Laravel Logs

If you don't use an error tracking service like Sentry or GlitchTip, Laravel 11 by default will log to the storage/logs/laravel.log file. You can access that by running a command like this:

# Find the service name of your application
sudo docker service ls

# Open a shell to the first container ID returned from your service
sudo docker exec -it $(sudo docker ps --filter "name=<service_name>" --format "{{.ID}}" | head -n 1) sh

# View the available log files
ls storage/logs/

# View the laravel.log file
cat storage/logs/laravel.log

You can see it's a bit of a process to get to the log file, so that's why we recommend using an error tracking service like Sentry or GlitchTip which you can self-host for no cost if you wanted.

Enabling Debug Mode

Depending on the service you're troubleshooting, the official documentation for the Docker image you're using may have instructions on how to enable debug mode.

For example, if you're running serversideup/php:8.4-fpm-nginx, the official documentation talks about setting LOG_OUTPUT_LEVEL=debug.

In this example, we would just send up a commit with the environment variable set.

services:
  php:
    image: ${SPIN_IMAGE_DOCKERFILE_PHP}
    environment:
      LOG_OUTPUT_LEVEL: debug

Services won't start or are not accessible

When services won't start or they are not accessible, they are usually broken down into two categories:

1. The container or service fail to start
2. The container or service is running, but you can't access it through HTTP/HTTPS

Types of Health Checks

When you're running applications with zero-downtime deployments, everything depends on health checks. There are multiple levels of health checks, so make sure you reference above to determine where your issue might be:

1. The Docker Swarm health check (the container or service fails to start)
2. The reverse proxy health check (the container or service is running, but you can't access it through HTTP/HTTPS)

Docker Health Check

The Docker health check is a health check that is more native to the container itself. Container authors can set HEALTHCHECK instructions in their Dockerfile to control the health of the container.

Health checks can also be overridden by the Docker Swarm service configuration at with the healthcheck option.

One of the most frustrating things might be you try using the docker service logs command, but the service doesn't have any logs because the service doesn't exist yet. To get around this, we can use the docker service ps command to get more information on why a container won't start.

sudo docker service ps --no-trunc <service_name>

The above command will show a glimpse of why a container won't start. If everything looks normal there, you might want to inspect the service with docker service inspect to see if there are any other issues.

sudo docker service inspect <service_name> --pretty

Adding the --pretty flag will show nice output of the service configuration and you can look for clues to see why the service might not be starting.

task: non-zero exit (137): dockerexec: unhealthy container

If you see an error message like the one above, this means something in the container is causing the Docker health check to fail. This could be caused by a start up script failing, a configuration issue, or something else.

Start up scripts can be tricky to debug because if any of the scripts send a "non-zero exit" code, the Docker health check could fail. Make sure you're properly handling exit codes and testing your scripts in all scenarios.

To get more information, put your container image in debug mode or add verbose logging and try again. You may get more information from docker service logs or docker service inspect.

Reverse Proxy Health Check

If you're trying to access your application through HTTP/HTTPS, but you're getting a 503 or 404 error, it's likely something is wrong with the reverse proxy health check. For our default configurations, we use Traefik as the reverse proxy, so we'll give you a few tips on how to proceed with troubleshooting.

services:
  php:
    image: ${SPIN_IMAGE_DOCKERFILE}
    environment:
      SSL_MODE: "full"
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.my-php-app.rule=Host(`${SPIN_APP_DOMA1N}`)"
        - "traefik.http.routers.my-php-app.entrypoints=websecure"
        - "traefik.http.routers.my-php-app.tls=true"
        - "traefik.http.routers.my-php-app.tls.certresolver=letsencryptresolver"
        - "traefik.http.services.my-php-app.loadbalancer.server.port=8080"
        - "traefik.http.services.my-php-app.loadbalancer.server.scheme=http"
        # Health check
        - "traefik.http.services.my-php-app.loadbalancer.healthcheck.path=/invalid"
        - "traefik.http.services.my-php-app.loadbalancer.healthcheck.interval=30s"
        - "traefik.http.services.my-php-app.loadbalancer.healthcheck.timeout=1s"
        - "traefik.http.services.my-php-app.loadbalancer.healthcheck.scheme=http"

Invalid Host

I have a typo in my Host rule: Host(${SPIN_APP_DOMA1N})"

Since the variable is SPIN_APP_DOMAIN, I have a typo and used SPIN_APP_DOMA1N instead. This would cause a 404 error because the host is invalid and the variable is undefined.

To fix this I would need to fix my typo and ensure SPIN_APP_DOMAIN is properly defined.

Invalid Port + Scheme

If I look at the documentation of serversideup/php, you can see I have SSL_MODE: "full" set in the service configuration. This tells the application to use HTTPS.

However, there are 4 issues with the configuration because of this change:

• server.port=8080: The docs say this should be changed to 8443 when SSL_MODE: "full" is set.
• server.scheme=http: Since we're using HTTPS, this should be set to https.
• healthcheck.timeout=1s: This health check timeout is quite low. This could cause a health check failure if the application takes longer than 1 second to start.
• healthcheck.scheme=http: The health check definition is different from the server definition. The health check should use https.
• healthcheck.path=/invalid: If I don't have a page at /invalid, this will cause a health check failure. I can read the serversideup/php docs to set this to /healthcheck to use the native container health check endpoint or /up if I am running Laravel.

Wow, you can see how many moving parts it takes to get zero-downtime deployments working. This process is true regardless of any service that you're running, so it's important to understand how each piece works together.

Always remember:

• Start with the service, check to make sure your application is actually getting deployed
• Then check the reverse proxy configuration.
• The health check MUST return a 200 status code to pass. Things like 301 or 302 redirects can cause the health check to fail.
• Always refer to the official documentation of the Docker image you're using for looking for proper health check definitions.

Database connection issues

A common thing we also see are people who have connection issues. This is usually caused by:

1. The database service isn't ready by the time a script in your other container tries to access it (controlled by container start orders).
2. The host is invalid, like localhost or 127.0.0.1, instead of using the name of the database service in their Docker Compose file (like postgres).
3. The credentials in the .env file get changed and the application can't connect to the database.

If you're provisioning a database like Postgres, MariaDB, or MySQL, they make an environment variable available called POSTGRES_USER andPOSTGRES_PASSWORD (or similar) that you can use to connect to the database.

To fix this, you'll need to reference the documentation of your database engine to manually change the password.

Diagnosing connection issues between services

If you notice connection errors between services, make sure your .env file has the correct values. Special things to double check:

• Ensure your DB_HOST (or comparable like REDIS_HOST) is set to the service name (ie. mariadb, redis, and NOT 127.0.0.1 or localhost)
• Ensure your environment variable values don't have any weird special characters in them.
• Ensure your APP_URL in .env.<environment> is set correctly (ie. https://app.example.com).
• Ensure any services have strong passwords set. Sometimes default passwords are rejected for security reasons. You can use a number of online tools to generate a strong password for your service.

Stopping a service

If you need to stop a service, you can use the docker service rm command. For example, to stop a service named my-service, you can run:

sudo docker service rm my-service

Replace my-service with the name of the service you want to stop.

Removing volumes

Volumes are used to persist data in your containers. If you need to remove a volume, you can use the docker volume rm command. For example, to remove a volume named my-volume, you can run:

sudo docker volume rm my-volume

Replace my-volume with the name of the volume you want to remove. You can find the volume name by running sudo docker volume ls.

Removing Swarm configurations

If you need to remove a Swarm configuration, you can use the docker config rm command. For example, to remove a configuration named my-config, you can run:

sudo docker config rm my-config

Replace my-config with the name of the configuration you want to remove. You can find the configuration name by running sudo docker config ls.

Removing networks

If you need to remove a network, you can use the docker network rm command. For example, to remove a network named my-network, you can run:

sudo docker network rm my-network

Pruning unused resources

If you need to remove unused resources, you can use the docker system prune command. For example, to remove all stopped containers, unused networks, and dangling images, you can run:

sudo docker system prune

This command removes all stopped containers, unused networks, and dangling images. You can add the --all flag to remove all unused images as well.

This will free up disk space on your server.

Starting over

We've all been there. The process of learning sometimes can start with a "redo". The good news is since everything is containerized, everything is designed to be disposable and repeatable.

Since we're running Docker, we do not need to rebuild the server. We can simply remove the Docker data and start over with a fresh deployment.

sudo docker service rm $(sudo docker service ls -q) # Remove all services
sudo docker stop $(sudo docker ps -aq) # Stop all containers
sudo docker rm $(sudo docker ps -aq) # Remove all containers
sudo docker system prune --all # Remove all unused images and networks
sudo docker volume rm $(sudo docker volume ls -q) # Remove all volumes
sudo docker config rm $(sudo docker config ls -q) # Remove all configurations

Run the commands above individually to ensure everything is removed properly. Once you've verified everything is removed, you can deploy a fresh copy of your application.

Re-provisioning servers from scratch

Sometimes you may want to completely delete a server from your cloud provider and start fresh with a new one. This is common when:

• You ran into issues during initial provisioning and want to start over
• You're testing your infrastructure setup
• You want to move to a different server type or location

When you delete a server from your cloud provider (like Hetzner, DigitalOcean, or Vultr) and want to re-provision, you'll need to clean up a few things first.

Step 1: Remove the IP address from .spin.yml

When Spin provisions a server through a provider, it automatically updates your .spin.yml file with the server's IP address. If you delete the server and want Spin to create a new one, you need to remove this address property.

servers:
  - server_name: ubuntu-2gb-ash-1
    environment: production
    hardware_profile: hetzner_2c_2gb_ubuntu2404
    address: 123.45.67.89  # Remove this line

servers:
  - server_name: ubuntu-2gb-ash-1
    environment: production
    hardware_profile: hetzner_2c_2gb_ubuntu2404

Step 2: Remove the server from SSH known_hosts

Your local machine stores SSH host keys for servers you've connected to. When you create a new server, it will have a different host key, which causes SSH to show a warning about a potential security issue.

To remove the old host key entry, run:

ssh-keygen -R your.server.ip.address

Replace your.server.ip.address with the IP address or hostname of the server you deleted. For example:

ssh-keygen -R 123.45.67.89

ssh-keygen -R myserver.example.com
ssh-keygen -R 123.45.67.89 # Remove the IP address entry

Step 3: Re-provision your server

Once you've cleaned up the .spin.yml file and SSH known_hosts, you can provision a fresh server:

spin provision

Spin will create a new server with your cloud provider and update your .spin.yml file with the new IP address.