After the blocked Store and Forward sessions were purged and the affected calculations were rebuilt, calculated events reached the historian and the calculations entered Live Mode. Follow the packet from the calculation engine to the target dataset: the decisive symptom is calculated output remaining in the Store and Forward buffer, not merely a calculation displaying Backfilling.
Where does the calculated event path stop?
The calculation engine evaluates source data, creates calculated events, and submits them through the Store and Forward service (SaF) for insertion into the historian. A calculation remains in Backfilling while its historical output is still being processed. Dependent calculations cannot start when their upstream inputs never complete that transition.
| Path stage | Reading to take | Interpretation | Next check |
|---|---|---|---|
| Source history | Confirm the required source time range exists | Missing source history prevents valid evaluation | Correct the calculation range or restore the required source data |
| Calculation engine | Check whether output events continue to be generated | Generation with no historian writes moves the fault downstream | Inspect the SaF session |
| Store and Forward | Watch the calculations session queue and write progress | A queue that retains calculated events without draining identifies the blocked hop | Inspect SaF errors and target tags |
| Historian target | Query the expected calculated tags over the backfill range | No new target events confirms that buffered inserts are not completing | Test for missing tags or deleted data files |
Layer one first: confirm that the services and historian endpoint are reachable before interpreting application state. If the SaF queue grows while connectivity remains healthy, the packet reaches SaF and stops during target-side insert or range maintenance.
Does the SaF session continue writing?
Observe the SaF session used by Calculations over a fixed interval. Record its queued-event count, last successful write, error state, and whether the target calculated tags receive new events. The evidence needed is movement: a large queue can be healthy if it drains, while an unchanged queue with no target writes is blocked.
| Queue and target reading | Meaning | Branch |
|---|---|---|
| Queue drains and target timestamps advance | SaF is operating; the calculation may still be processing a large historical range | Continue monitoring calculation progress |
| Queue grows and no target timestamps advance | Events reach SaF but inserts do not complete | Read the SaF error and inspect target existence |
| No queued events and no calculated output | The stop occurs before SaF | Check source history, expression inputs, and calculation service diagnostics |
| Other SaF sessions write but the calculations session does not | The fault is isolated to the calculation session or its target operations | Prepare session-specific recovery |
Restarting the calculation, calculation service, or server does not remove a persistent operation already held in the buffer. Deleting and recreating only the calculation can also leave the blocked SaF session untouched. That explains why all four restart or recreation attempts can return to Backfilling.
Does the target tag or historical range still exist?
Read the SaF error associated with the first non-advancing operation. The observed failure occurred while SaF attempted to remove old data and insert the backfill after underlying data files had already been deleted. The queued operation expected target data to exist, so range deletion for a nonexistent tag or range blocked the session.
Compare the buffered operation with the historian catalog and storage state. Confirm that the target calculated tag exists, that its target dataset is present, and that the requested backfill range maps to available source and target storage. If virtual views expose these tags, check their mappings separately; a view can be rebuilt correctly only after the dataset and tags exist.
| Reading | Outcome | Action |
|---|---|---|
| SaF reports a delete-range failure for a target that does not exist | The known blocking condition is present | Use the blocked-session recovery branch |
| Target exists but source history is absent | The requested output cannot be recalculated from that start time | Select a valid start time or restore source history |
| Target and source history exist, but inserts fail | A different storage or session error controls the fault | Resolve the exact SaF error before purging anything |
| SaF writes normally | The buffer is not the stopping point | Return to calculation diagnostics |
Which recovery branch applies?
Preserve the diagnostic state before changing the queue. Record the calculation names, original backfill start times, target dataset, virtual-view mappings, SaF session identity, and first blocking error. Purging a session discards its queued operations, so treat it as a controlled recovery action and retain the information required to regenerate calculated output.
For the observed blocked-session case, purge the offending SaF sessions, then rebuild each calculation from a copy. Copying allowed each replacement calculation to start from the original backfill time. Delete the original only after capturing its expression, inputs, output configuration, and start time. Rebuild the target dataset and virtual views where deletion or recreation removed those objects.
Canary version 24.1.1 contains a bug fix for a similar condition identified as Store and Forward: Will block session when deleting range for tag that does not exist. Place an upgrade to 24.1.1 in the recovery plan when the installed version predates that fix. Verify the installed version directly rather than inferring it from the symptom.
How do you recover the blocked calculation chain?
- Stop changes to the affected calculation chain so that names, dependencies, target tags, and start times remain traceable.
- Capture the SaF error, queued-session state, calculation definitions, target dataset configuration, and virtual-view mappings.
- Confirm that calculated events are held in the calculations SaF session and that the target historian tags are not advancing.
- Purge only the offending sessions through the supported SaF administration workflow. Do not purge unrelated sessions that continue to write.
- Create a copy of the first upstream calculation and configure it with the original backfill start time.
- Remove the original calculation after its copied definition has been checked field by field.
- Rebuild the required target dataset and virtual views, then start the copied calculation.
- Repeat the copy, delete, rebuild, and start sequence in dependency order. Complete upstream calculations before starting calculations that consume their outputs.
- Apply the
24.1.1upgrade through the site change-control process when the installed version lacks the cited fix.
What readings prove the fix?
Verification must cover the whole path. First, watch the calculations SaF session and confirm that its queued-event count decreases and successful writes resume. Next, query the target calculated tags at the original backfill start time, at points inside the range, and near the current time. Values must appear in chronological order without a gap created by the purge.
Confirm that the upstream calculation leaves Backfilling and enters Live Mode. Start each dependent calculation only after its input tags contain the required historical range. Finally, create or execute a controlled backfill against a valid target, verify that SaF processes any range replacement without blocking, and confirm that the target tag advances into Live Mode.
FAQ
How do I tell whether Canary Calculations or Store and Forward is stuck?
Check whether calculated events enter the calculations SaF session and whether its queue drains. Events held there while target tag timestamps remain unchanged place the stop in SaF or the historian insert path.
How do I clear calculations stuck in Backfilling?
Capture the session error and calculation configuration, purge only the offending SaF sessions, then copy and rebuild the calculations from the original backfill time in dependency order. Rebuild the target dataset and virtual views where required.
How do I prevent a deleted tag from blocking backfill?
Verify that each target tag and dataset exists before starting range deletion and backfill. Canary 24.1.1 includes a fix for a similar session-blocking condition involving deletion of a range for a tag that does not exist.
How do I verify a rebuilt calculation is in Live Mode?
Confirm that the SaF queue drains, target values exist from the original backfill start time through the current range, and upstream calculations enter Live Mode before their dependents start. Finish by running a controlled valid backfill and verifying that its target timestamp advances into Live Mode.