Telegraf Kubernetes Restarts: Troubleshooting Probes

Telegraf 1.12.6 was not crashing while writing to InfluxDB. The log shows a successful write followed by an orderly shutdown, while the Kubernetes Deployment used HTTP liveness and readiness probes with a 1-second timeout. Increasing the probe tolerance and timeout stopped the restarts.

Identify the shutdown mechanism

The webhook request completed its full data path: Telegraf accepted the Particle-format JSON, wrote one metric to InfluxDB, and made the data visible to the dashboard. The InfluxDB output reported a successful write in 4.584854 ms, with an empty buffer afterward.

That timing alone does not prove which component initiated termination, but it directs the investigation toward external lifecycle management and health checks.

Check the Kubernetes probe configuration

The container exposed the Telegraf health output on port 8888. Kubernetes sent both liveness and readiness requests to / on that port with the following settings:

livenessProbe:
  failureThreshold: 3
  httpGet:
    path: /
    port: 8888
    scheme: HTTP
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1

readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /
    port: 8888
    scheme: HTTP
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1

The confirmed resolution was to make these probes less aggressive by increasing the allowed successful attempts and increasing the timeout. The evidence does not provide the final numeric settings, so select them from measured health-endpoint response behavior instead of copying an unsupported value.

Review what the health endpoint evaluates

The health output listened on http://:8888 and evaluated buffer_size with two conditions:

[[outputs.health]]
  service_address = "http://:8888"

[[outputs.health.compares]]
  field = "buffer_size"
  lt = 5000.0

[[outputs.health.contains]]
  field = "buffer_size"

Both conditions must be true. Without metric filtering, the checks are not restricted to the InfluxDB writer series. One proposed correction was to scope health evaluation to the InfluxDB internal-write metric:

namepass = ["internal_write"]
tagpass = { output = ["influxdb"] }

This filtering requires the internal input to be enabled. Treat the broad health condition as a contributing hypothesis unless probe results confirm that the endpoint returned an unhealthy response during the write; the demonstrated fix was increased Kubernetes probe tolerance.

Apply the troubleshooting procedure

1. Reproduce the issue with one POST and confirm whether InfluxDB logs a successful batch before shutdown.
2. Distinguish an orderly stop from a crash. The sequence Stopping service inputs, cache flushing, output closure, and Stopped Successfully indicates controlled termination.
3. Increase probe timeout and attempt tolerance. Do not change the webhook payload merely because the restart follows a POST; the same behavior occurred with both the Particle webhook and Python sender.
4. If the health output should represent only InfluxDB writer health, enable the internal input and apply the supported namepass and tagpass filters.

Verify the correction

Send repeated webhook requests through port 1619 at path /particle. Verify that Telegraf continues listening, each batch reaches the telegraf database at http://influxdb.monitoring:8086, and Kubernetes does not restart the container. Also confirm that the health endpoint remains responsive through write activity and that the log no longer enters the controlled shutdown sequence after a POST.

FAQ