Troubleshooting FT Optix Sorting Machine Tracking Guide

Brian Holt7 min read
Allen-BradleyHMI / SCADATroubleshooting
Licensed PE Working through this on a live machine? A Maine-licensed engineer can take it from here — included with IMD hardware, by the hour for everything else. Book an engineer

Samples that appear twice, drift out of position, or exchange places point to separate event-handling failures. Restore production by recording the exact sequence, accepting each sensor passage once, and allowing each FIFO operation through one interlocked transaction. Debounce alone cannot provide those guarantees.

Prove the failure before changing the logic

Start with the usual quick fixes and rule them out. Increasing the debounce time may suppress electrical chatter, but it does not stop level-driven logic from executing on multiple scans. Rechecking the queue code offline will not expose two routines acting in the wrong order. Leaving a large logger active continuously can also create a second problem: polling multiple large arrays of UDT instances drove disk and network use high enough to damage FT Optix application performance.

Observed symptom Likely event failure Record needed
One physical sample becomes two records The entry event is accepted more than once Raw sensor, debounced state, acceptance event, and queue load
All later samples are offset An extra or missing queue load changed the count Queue count before and after every load
Two records exchange order A transfer executed twice, used the wrong source position, or wrote destinations in the wrong order Source and destination identities before and after the transfer

Keep printer, ArmorBlock, and Transaction Manager faults on separate fault timelines until an event record connects them to tracking. Check this stage by capturing one failure with enough state to classify it as duplicate entry, count offset, or order reversal.

Build a logger the machine can carry

Use FT Optix as a targeted event recorder, not as an unrestricted array poller. Simple Data Logger configuration is workable for small tag sets, but multiple large UDT arrays required scripted construction with C# and NetLogic in this application. The logger also needs a runtime enable and disable path so troubleshooting load does not become a permanent production load.

  1. Select only the signals that establish causality: raw and debounced entry states, the accepted-entry event, every FIFO load or transfer request, queue counts, and the record identities at affected boundaries.
  2. Add an internal sequence identity to each accepted physical arrival. The printed sample number is insufficient because valid samples arrive in identical pairs.
  3. Capture values when an entry or transfer event occurs. Avoid repeatedly polling every field of every queue when a small event snapshot answers the question.
  4. Provide an operator- or maintenance-controlled logging enable. Keep its state visible so nobody diagnoses a run without knowing whether capture was active.
  5. Separate diagnostic retention from live visualization. The display does not need to redraw every recorded array element.

Compare application responsiveness plus disk and network activity with logging disabled and enabled. Proceed only when the capture runs without materially changing the machine or FT Optix behavior.

Accept each entry sensor passage once

The first sensor sometimes detected the same sample twice even though its input was debounced. Because identical sample numbers legitimately arrive in pairs, the second detection passed as believable data, invented another sample, and shifted all later tracking.

Convert the sensor signal into a qualified event. Accept only a debounced transition while the entry sequence is armed and in the correct machine state. Immediately disarm the acceptance path after loading the record. Rearm it only after the sensor returns clear and the entry sequence is ready for another physical sample.

  1. Observe the raw input and confirm whether it changes twice or remains active across multiple scans.
  2. Compare the debounced state with the actual queue-load event. One stable input that produces two loads identifies a logic qualification problem rather than contact bounce.
  3. Gate the accepted event with both edge detection and sequence state.
  4. Block every alternate routine from loading the same arrival.
  5. If machine state cannot distinguish a new sample from the previous sample, add another sensing condition and compare the two discrete states before accepting the arrival.

Pass samples individually and as valid identical pairs. The check passes when each physical sensor passage produces exactly one accepted sequence identity and one queue load.

Interlock every FIFO load and transfer

A FIFO preserves order only when the surrounding logic requests each operation exactly once. It cannot correct a premature load, a delayed load, or two loads during one process window. Treat every load and queue-to-queue movement as a transaction with ownership, qualification, execution, and acknowledgment.

Guard Purpose Failure it blocks
Valid process window Permits the operation only at the intended machine state Pre-load or post-load
One event owner Prevents parallel routines from commanding the same queue Competing writes
Single execution latch Allows one operation until completion is acknowledged Multiple loads from a sustained trigger
Source-valid check Confirms a record exists before transfer Empty or stale movement
Destination-ready check Confirms the receiving queue can accept the record Partial or overwritten transfer

Review every place that can load, unload, shift, or transfer a queue. Include bit shifts or flips if they participate in position tracking. Record request, execution, and acknowledgment separately; a request remaining true must not cause repeated execution.

Check the event trace from entry through the first transfer. Every accepted sequence identity must have one load and every completed movement must change the source and destination counts once.

Find the first boundary that reverses order

An offset and a swap are different faults. An extra entry normally shifts everything after it. A swap means two identities were correct before a boundary and reversed after it. Search for the first boundary where that invariant fails instead of inspecting the entire sorting sequence at once.

  1. Capture the ordered identities at the source queue immediately before a transfer.
  2. Capture the selected source position and destination position used by the operation.
  3. Record every condition that can request the movement during that process window.
  4. Capture both queues immediately after completion.
  5. Compare the before-and-after order using the internal sequence identity, not the duplicated sample number.

If order is already wrong before the transfer, move the trace point upstream. If it changes at the transfer, inspect execution order, repeated triggers, overlapping source and destination writes, stale position selection, and any shift operation associated with that boundary. Move the entire record as one controlled transaction; do not let another routine modify either queue midway through the move.

The boundary passes when repeated controlled transfers preserve sequence identity and count, including transfers of two records carrying the same sample number.

Run the complete tracking proof

Get it running, then fix it properly. Use the runtime logger for fault isolation, then leave behind the entry qualification and transfer interlocks that prevent recurrence.

  1. Run single samples and confirm one sensor passage, one accepted identity, and one initial load.
  2. Run valid identical pairs and confirm that the internal identities remain distinct and ordered.
  3. Trace every queue boundary and prove that each transfer conserves record count.
  4. Compare final sorting order with entry order and the commanded routing decisions.
  5. Disable diagnostic logging and repeat the run to confirm normal FT Optix disk, network, and application performance.
  6. Retain the failing and passing event records with the controller logic and FT Optix project revisions used for the test.

Production verification passes only when physical arrivals equal accepted entries, each operation executes once, queue counts balance across transfers, and sequence identities never reverse without an intentional routing decision.

FAQ

Why does a debounced sensor still count one sample twice?

Debounce filters input transitions; it does not stop level-driven logic from loading on multiple scans. Generate one qualified event, disarm it after the load, and rearm only after the sensor clears and the sequence is ready.

Why does FIFO tracking invent an extra sample?

An entry trigger can execute before, after, or more than once within its valid process window. Log the accepted-entry event beside the queue load and require exactly one load per internal sequence identity.

Why does a sorting machine swap two samples?

Find where two sequence identities are ordered correctly before a queue boundary and reversed afterward. At that boundary, check for repeated transfers, wrong position selection, overlapping writes, and shift logic executing out of order.

Why does FT Optix logging slow the application?

Polling multiple large arrays of UDT instances can drive disk and network use sharply upward. Capture a small event-based signal set and provide runtime controls to disable the logger outside diagnostic runs.

Stop here if enabling the logger destabilizes runtime operation, device faults prevent a controlled test, or a dry run still changes queue order without a recorded command. Preserve the event capture, project revisions, and device diagnostics, then escalate through the official product support channel.

Back to blog