DepthAI Watchdog: Troubleshooting Reconnection Timeout

Daniel Price4 min read
Industrial NetworkingOther ManufacturerTroubleshooting
Licensed PE Working through this on a live machine? A Maine-licensed engineer can take it from here — included with IMD hardware, by the hour for everything else. Book an engineer

A disconnected DepthAI device can spend substantially longer in failure detection than in the visible reconnection attempt. The supplied logs show a fixed 10000ms reconnection message even after setting connection-related environment variables and dai.BoardConfig watchdog fields. The evidence does not identify a supported setting that changes this host reconnection timeout, so treat watchdog timing, initial connection timing, failure detection, and reconnection timing as separate controls.

Separate the timeout mechanisms

Mechanism or setting Evidence Engineering conclusion
watchdogInitialDelayMs and watchdogTimeoutMs The script sets these through dai.BoardConfig, but the RVC4 test still takes more than two minutes. Do not assume board watchdog fields control host-side disconnection detection or reconnection duration.
DEPTHAI_CONNECT_TIMEOUT It is set to 2000, while the log still reports Attempting to reconnect. Timeout is 10000ms. The test does not demonstrate control of the logged reconnection timeout.
DEPTHAI_WATCHDOG, DEPTHAI_WATCHDOG_INITIAL_DELAY, and DEPTHAI_BOOTUP_TIMEOUT These lines are commented out in the supplied script. This run cannot establish their effect. Earlier attempts reportedly did not change the behavior, but no confirming log is supplied.
setMaxReconnectionAttempts(1, reconnection_callback) After one unsuccessful attempt, the callback prints the failure message and exits. This limits the number of retries; it does not prove control over failure-detection or per-attempt timeout.

Identify where the RVC4 delay occurs

In the OAK4 log, the monitor reports a missed ping at 12:09:44.944. The DeviceGate failure is not reported until 12:11:59.630, a derived interval of . Reconnection then runs from 12:11:59.630 until 12:12:10.557, approximately . The total derived interval from the missed-ping message to the failed callback is therefore approximately .

This breakdown shows that changing only the displayed 10000ms reconnection interval could not meet an exit target of six to eight seconds in that run. Most of the delay occurred before reconnection began. The evidence does not reveal which supported RVC4 setting, if any, controls that earlier DeviceGate interval.

Compare the observed device paths

Test Failure-detection behavior Reconnection behavior Observed result
OAK4 Missed-ping warning precedes DeviceGate failure by . Log reports 10000ms; failed attempt lasts approximately . More than two minutes before exit.
OAK-FFC-4P No comparable long DeviceGate interval appears in the supplied test log. From connection closure to unsuccessful reconnection is approximately . Approximately 12 seconds before exit.

The logs support different failure-detection paths, but they do not establish that the same watchdog facility or timeout variable applies to both device architectures. Do not infer RVC4 support from an RVC2 watchdog description.

Build a controlled timeout test

  1. Set environment variables before creating the pipeline or device, as the supplied script already does.
  2. Test one candidate setting at a time. Record the first communication or missed-ping timestamp, the Closed connection timestamp, the Attempting to reconnect timestamp, and the callback timestamp.
  3. Keep setMaxReconnectionAttempts(1, reconnection_callback) while testing so repeated successful-but-unstable reconnections do not obscure the timing of one attempt.
  4. Repeat the test separately on each device architecture. A result from the OAK-FFC-4P path does not verify RVC4 behavior.
  5. Accept a setting only if the relevant measured interval changes. A shorter initial connection timeout does not count as proof that disconnection detection or reconnection changed.
  6. If no documented control changes the long pre-reconnection interval, enforce the application deadline outside the blocked device operation so the Docker supervisor can terminate and relaunch the process. The supplied evidence does not define a DepthAI setting that guarantees the requested three-second or six-to-eight-second exit.

Correct and verify the exit path

The callback currently calls os._exit(1), which terminates immediately without normal Python cleanup. That is consistent with forcing a container restart, but it also bypasses context-manager cleanup and buffered output. Use it only when immediate termination is intentional.

The exception handlers should also be reordered because RuntimeError is handled by the preceding broad Exception clause:

except RuntimeError as e:
    print(f"Exiting Program due to [RuntimeError]: {e}")
    os._exit(1)
except Exception as e:
    print(f"Exiting program due to [Exception]: {e}")
    os._exit(1)

Verify the final design by measuring from the chosen disconnection indicator to actual process termination. For the OAK4 case, separately verify the pre-reconnection and reconnection intervals; otherwise a fixed 10000ms retry can hide a much larger DeviceGate delay.

FAQ

Why does DEPTHAI_CONNECT_TIMEOUT=2000 not change the 10000ms reconnection timeout?

In the supplied test, DEPTHAI_CONNECT_TIMEOUT is 2000 but the log still reports a 10000ms reconnection timeout. The evidence therefore does not show that this variable controls reconnection after an established device disconnects.

Why does the OAK4 process take more than two minutes to exit?

The derived interval between the missed-ping warning and DeviceGate failure is , followed by approximately of reconnection. The delay is primarily before the reconnection attempt.

Can setMaxReconnectionAttempts make DepthAI exit within three seconds?

setMaxReconnectionAttempts(1, ...) limits retries, but the logs do not show that it shortens failure detection or the 10000ms attempt. Meeting a three-second deadline requires control of every preceding interval or an external process deadline.

Back to blog