Skip to content

Auto-restart

DUMB includes an automatic restart system that monitors service health and restarts failed services to maintain system stability without manual intervention.

Overview

The auto-restart system provides:

  • Health monitoring - Periodic health checks for each service
  • Automatic recovery - Restart services that become unhealthy
  • Exponential backoff - Increasing delays between restart attempts
  • Restart limits - Prevent infinite restart loops
  • Grace periods - Allow services time to initialize

Auto-restart status Auto-restart indicators

How it works

%%{ init: { "flowchart": { "curve": "basis" } } }%%
flowchart TD
    A([Service running])
    B{Health check}
    C{Threshold exceeded?}
    D{Restart limit reached?}
    E[Restart service]
    F([Stop retrying])
    G[Wait grace period]

    A ==> B
    B -- Healthy --> A
    B -- Unhealthy --> C
    C -- No --> B
    C -- Yes --> D
    D -- Not reached --> E
    D -- Reached --> F
    E ==> G
    G ==> B
  1. Health check - Service is periodically checked for responsiveness
  2. Unhealthy detection - Multiple consecutive failures trigger action
  3. Restart attempt - Service is stopped and restarted
  4. Grace period - Wait for service to initialize
  5. Repeat - Continue monitoring after restart

Configuration

Auto-restart is configured globally in dumb.auto_restart:

"dumb": {
  "auto_restart": {
    "enabled": false,
    "restart_on_unhealthy": true,
    "healthcheck_interval": 30,
    "unhealthy_threshold": 3,
    "max_restarts": 3,
    "window_seconds": 300,
    "backoff_seconds": [5, 15, 45, 120],
    "grace_period_seconds": 30,
    "services": []
  }
}

Configuration options

Option Default Description
enabled false Enable auto-restart globally
restart_on_unhealthy true Restart when health checks fail
healthcheck_interval 30 Seconds between health checks
unhealthy_threshold 3 Consecutive failures before restart
max_restarts 3 Maximum restarts within the window
window_seconds 300 Time window in seconds
backoff_seconds [5, 15, 45, 120] Backoff delays between restarts
grace_period_seconds 30 Seconds to wait after restart before health checks
services [] Limit auto-restart to these process names

Exponential backoff

To prevent rapid restart loops, delays between restarts increase exponentially:

Attempt Delay
1 5 seconds
2 10 seconds
3 20 seconds
4 40 seconds
5 80 seconds
6+ 120 seconds (max)

The formula: delay = min(initial_delay * (backoff_multiplier ^ attempt), max_delay)

Restart limits

Services have a maximum number of restart attempts within a time window:

  • Default: 5 restarts per hour
  • After reaching the limit, auto-restart pauses for that service
  • The counter resets after the window expires
  • Manual restart resets the counter

Restart limit reached

If a service keeps failing, investigate the root cause rather than increasing limits. Check logs for error messages.

Health checks

Services are monitored using health check endpoints or process status:

HTTP health checks

For services with web interfaces:

"health_check": {
  "type": "http",
  "url": "http://127.0.0.1:8080/health",
  "timeout": 10,
  "interval": 30
}

Process health checks

For services without HTTP endpoints:

"health_check": {
  "type": "process",
  "interval": 30
}

Monitoring restart status

Dashboard indicators

The dashboard shows auto-restart status for each service:

  • Restart count - Number of restarts in current window
  • Last restart - Timestamp of most recent restart
  • Health status - Current healthy/unhealthy state

API endpoints

Query restart status via the API:

# Get service status including restart info
curl http://localhost:8000/api/process/service-status?process_name=Riven%20Backend

Response includes:

{
  "process_name": "Riven Backend",
  "status": "running",
  "healthy": true,
  "restart": {
    "count": 2,
    "last_restart": "2025-01-15T10:30:00Z",
    "enabled": true
  }
}

WebSocket updates

Real-time restart events via /ws/status:

{
  "type": "status",
  "processes": [
    {
      "process_name": "Riven Backend",
      "status": "running",
      "healthy": true,
      "restart": {
        "count": 2,
        "last_restart": "2025-01-15T10:30:00Z",
        "enabled": true
      }
    }
  ]
}

Disabling auto-restart

Per-service

Disable for a specific service:

"riven_backend": {
  "auto_restart": {
    "enabled": false
  }
}

Globally

To disable auto-restart for all services, set enabled: false in each service's auto_restart configuration, or use the Settings page in the frontend.

Best practices

Appropriate thresholds

  • Critical services (Plex, rclone): Lower threshold (2-3)
  • Background services (Zilean, NeutArr): Higher threshold (3-5)

Grace periods

  • Fast-starting services: 10-15 seconds
  • Database-dependent services: 30-60 seconds
  • Services with startup tasks: 60-120 seconds

Monitoring

  • Review restart counts regularly
  • Investigate services with frequent restarts
  • Check logs after restart events

Troubleshooting

Service keeps restarting

  1. Check service logs for errors
  2. Verify configuration is valid
  3. Ensure dependencies are running
  4. Check for port conflicts

Auto-restart not working

  1. Verify auto_restart.enabled is true
  2. Check if restart limit was reached
  3. Ensure health check is configured correctly

Restart delay too long

  • Reduce backoff_multiplier
  • Lower max_delay
  • Reset counter with manual restart