How to detect a bot that is running but no longer receiving data, evaluating strategies or reaching its broker API.
In plain English
This guide focuses on what happens after the code is deployed: how to know it is alive, whether it is still doing useful work and how to recover when it is not.
health checkheartbeatstructured logsrecovery
Practical example
A health check should answer a real question
“Process exists” is weak. A stronger check can prove that the service loop is recent, required data is fresh and the external API session can still be used.
Four timestamps beat one “alive” flag
last_loop_at
last_market_data_at
last_strategy_run_at
last_broker_success_atEach timestamp represents a different dependency. If only last_loop_at is fresh, the process may be alive but disconnected from its data source.
Simple watchdog rule
if now - last_market_data_at > 60:
alert("market_data_stale")
if now - last_broker_success_at > 300:
alert("broker_api_stale")Do not alert on expected quiet periods
Market closures, scheduled maintenance and intentionally paused trading need separate states. Otherwise monitoring becomes noisy and operators start ignoring real alerts.
Before you rely on this in production
- Heartbeat/health data has a freshness threshold.
- Alerts are actionable rather than noisy.
- Logs include correlation/event IDs.
- Recovery behavior is documented and tested.