Resolving Ignition Custom Tag History Aggregate Min/Max Errors

Which hops does the aggregate query cross, and where does it stop?

Follow the packet. A custom Python aggregate referenced as shared.myMinMax does not run where you call it. The Designer Script Console sends a tag-calculation request to the gateway. The gateway pulls history from the provider (here [Canary/default:default]), runs the Python function once per value, serializes the result dataset as XML, and ships it back. The client then parses every cell against the column type the gateway declared. The stack trace maps onto those hops exactly:

Hop	Component	Frame in the trace	What happens
1	Designer Script Console (Jython)	JythonConsole$ConsoleWorker.doInBackground	system.tag.queryTagCalculations is called
2	Client tag utilities	ClientTagUtilities.queryTagCalculationsImpl	Builds the calculation request
3	Gateway interface, outbound	GatewayInterface.runTagCalculationQuery / sendMessage	Request sent to the gateway
4	Gateway: history provider + aggregate engine	Not in the client trace	Provider returns values; myMinMax executes gateway-side
5	Gateway interface, inbound	GatewayInterface.getResponse → SAX parser	XML response arrives and is parsed
6	Response parser	ResponseParser.endElement → TypeUtilities.coerceLocaleSafe → Long.parseLong	Stops here: NumberFormatException: For input string: "299748.8405131454"

The outer message, GatewayException: Unable to read response from Gateway, is a wrapper. Read the innermost Caused by first; that is the layer-one reading for this class of fault. The innermost cause shows the gateway completed the query and returned data. The request died on the way back, in the client, while coercing a cell string into the declared column type. The branch you take depends on what that innermost cause says:

Innermost cause / result	Meaning	Next check
NumberFormatException in ResponseParser, column="AD-HOC-PYTHON", type=Long	Aggregate ran; declared column type does not match the value	Check 2 (column type)
Python exception (ImportError, NameError, TypeError)	Script failed on the gateway	Check 7 (gateway-side logging and imports)
No exception, one value per row where you expected two	Transport is fine; the return shape is the problem	Check 3 (multi-value return)
No exception, None or 0 every window	Seed or filter logic	Checks 4 and 5
No exception, values plausible but drift window to window	State carried across windows	Check 6

Check 2: What type did the result column declare, and what value arrived?

Take two readings from the parser frame: the declared type (type=Long) and the offending string ("299748.8405131454"). A dataset column carries one type for every row. The response is XML text, so each cell arrives as a string and is coerced to the column's type. Long.parseLong accepts digits and a sign only; the decimal point in a totalizer reading makes it throw. The failure is in row=0, so the very first window already mismatched.

Follow the value back to its source. With the original seed logic (Check 4), the function returned None on every window, yet row 0 carried a fractional totalizer reading in a Long column. The inference: when the script returns nothing usable or a multi-element list, the aggregate engine fills and types the cell itself, and the declared type does not follow the float values the script handled. An inline test that returned the integer list [400,500] produced cells typed Long (400L), which shows that integer returns surface as Long.

Outcome	Meaning	Action
Column Long, value fractional	Return type and data type diverged	Return one explicit float (or None) per window; go to Check 3
Column Double/Float, parse succeeds	Typing is fine	Continue to logic checks (4-6)
Column Long, values whole numbers	Silently truncated semantics if data later goes fractional	Cast returns to float anyway

Casting every non-empty return with float() is defensive: the result type then stays constant regardless of whether the provider hands back an integer-valued or fractional reading for a given window.

Check 3: Does the aggregate return more than one value per window?

The original function returns [current_min, current_max] when finished is true. The expectation is a Min column and a Max column, the way the built-in MinMax mode yields two values. Take the reading: count the columns and the values per row.

• Result columns from queryTagCalculations are [tagpath, AD-HOC-PYTHON]: one column for the custom calculation.
• A revised min/max function that ran without error on a running totalizer over one month returned a single value per row (153553.197439), not two.
• Returning system.dataset.toDataSet(['min','max'], [[current_min, current_max]]) from the aggregate produced the same single-value output. Custom column headers are not honored.
• An inline test through queryTagHistory with returnSize=2 and a function returning [400,500] returned [timestamp, 400L, timestamp, 400L]: two windows, first list element only.

Per the platform code, a returned list is packed as an array, and a list of tuples (value, qualityCode) with an old-style integer quality code is also accepted. The observed result is still one value per window: only the first element survives to the dataset. Built-in calculations are not affected. A call with calculations=['Average','Minimum','Maximum','StdDev'] returns one column per calculation without issue.

Decision: one custom aggregate returns one scalar per window. For min and max, write two aggregates and request both in one call (resolving branch below). Test return-shape behavior on your gateway with the inline harness before building on multi-value returns:

script = """\
python:def wrapper(qval, interpolated, finished, blockContext, queryContext):
    if finished:
        return [400, 500]
"""
ret = system.tag.queryTagHistory(tagPath, rangeHours=8,
                                 aggregationModes=[script], returnSize=2)
print [ret.getValueAt(r, c) for r in range(ret.rowCount) for c in range(ret.columnCount)]
# One value per window after each timestamp = first element only.

The python: prefix lets you pass the aggregate body as a string, so you can iterate without saving the project library each time.

Check 4: Is the minimum seeded so a real value can ever win?

The original initialization is blockContext.getOrDefault('current_min', float(0)). A totalizer never goes negative, so min(0.0, reading) always stays 0.0. The later guard if current_min == 0 or current_max == 0: return None then fires on every window. The function returns None everywhere, independent of the data.

Seed for min	Behavior on non-negative data	Verdict
0.0	Min pinned at 0; zero-filter defeated; returns None	Wrong
Largest float (sys.float_info.max)	First valid value replaces it	Works; leaks the sentinel if no valid value arrives
None sentinel	First valid value assigned directly; empty window returns None	Preferred: no magic number in the output

Seed the maximum the same way. 0.0 works for max on non-negative data, but a None sentinel keeps both functions symmetric and makes an all-zero window come back empty instead of reporting 0 as a maximum.

Check 5: Does the filter drop the zeros, and only the zeros?

Read the condition character by character. The original uses qval.quality.isGood() and qval.value != 0, which is correct. A later rewrite flipped it to qval.value == 0. That version accumulates only the erroneous zeros, the opposite of the intent. Another working form uses truthiness, if qval.quality.good and qval.value:, which rejects 0, 0.0, and None in one test.

Two more paths still let bad samples through:

• Quality. ignoreBadQuality=True on the call filters bad-quality rows before they reach the function. Keep the quality.good test in the script anyway, so the aggregate stays correct if someone calls it without that flag.
• Interpolated values. The interpolated argument marks values the engine synthesized, typically at window boundaries. If a stored spurious zero sits next to a real reading of 299748, an interpolated value between them is nonzero and passes a != 0 test. It then drags the minimum down to a number that was never recorded. For a totalizer with dropout zeros, skip samples where interpolated is true, or disable interpolation on the call. A working built-in call passes nointerpolation=0; set it true for this use.

If the minimum is still suspiciously low after the fix, pull raw rows for that window (fallback section) and look for a zero-adjacent sample. That reading decides whether interpolation is the leak.

Check 6: Is state carried in the right context across blocks and windows?

The function gets two dictionaries with different lifetimes. Mixing them changes what each row means.

Store	Lifetime	Use it for
blockContext	One aggregation window; the call with finished=True closes it	Per-window min/max
queryContext	Entire query, across all windows	Running extremes across windows, or stitching a window the engine processes in more than one block

One pattern seeds each block from the query-level extremes: blockContext.get('current_min', queryContext.get('lastMin', ...)), and writes back with queryContext['lastMin'] = min(current_min, lastMin). This guards against the engine splitting a single requested window into several blocks. The cost: with returnSize=100, row n becomes the min/max from the query start through window n, a running extreme rather than a per-window value.

• Whole-range answer, one row (no returnSize on queryTagCalculations): seed from queryContext.
• Independent per-window values (returnSize > 1): use blockContext only.

Also note the ordering in that pattern: when finished is true it returns before writing blockContext, which is correct because the block is discarded. Write blockContext on every non-final call, or the next value starts from the default seed again.

Check 7: Where does the debug output go, and does the library import?

The function executes on the gateway, so print inside it does not reach the Designer Script Console that issued the query. Its output lands in gateway-side standard output, if anywhere. Route diagnostics through a named logger, or through the logging functions exposed on queryContext:

logger = system.util.getLogger("shared.myMinMax")
logger.infof("Current Min: %.2f, Current Max: %.2f", current_min, current_max)

Watch the format token: %2f sets a field width of 2 with default precision, while %.2f gives two decimals. The function runs once per sample, so an info-level line per call on a 20-day totalizer floods the gateway log. Log at debug level and raise the logger's level only while diagnosing, or log once per window inside the finished branch.

Import pitfall: import sys.float_info as floatInfo fails with ImportError. float_info is an attribute of sys, not a submodule. Use import sys and reference sys.float_info.max, or use a None sentinel and skip the import. A module-level import error breaks every function in the shared library, not just the aggregate, so check the gateway log for it before debugging logic.

The float() casts in the original loop are unnecessary for comparison, because the provider delivers consistent numeric types. Keep the cast only on the returned value, per Check 2.

Resolving branch: how do you build nonzero min and max as two aggregates?

Split the calculation so each aggregate returns one scalar float per window, filters zeros and interpolated samples, and uses None for empty windows.

1. Open the gateway-scoped project library that holds shared and replace myMinMax with the two functions below.
2. Choose the context pattern from Check 6. The code below is per-window (blockContext only).
3. Save the project so the gateway picks up the library change.
4. Call both calculations in one query and read the columns by index.

# Project library: shared
logger = system.util.getLogger("shared.historyAgg")

def _usable(qval, interpolated):
    # Reject bad quality, None, 0 / 0.0, and engine-interpolated samples
    return (not interpolated) and qval.quality.good and bool(qval.value)

def myMin(qval, interpolated, finished, blockContext, queryContext):
    cur = blockContext.get('mn')
    if _usable(qval, interpolated):
        v = qval.value
        cur = v if cur is None else min(cur, v)
    if finished:
        return None if cur is None else float(cur)
    blockContext['mn'] = cur

def myMax(qval, interpolated, finished, blockContext, queryContext):
    cur = blockContext.get('mx')
    if _usable(qval, interpolated):
        v = qval.value
        cur = v if cur is None else max(cur, v)
    if finished:
        return None if cur is None else float(cur)
    blockContext['mx'] = cur

Call site:

end   = system.date.now()
start = system.date.addDays(end, -20)
path  = '[Canary/default:default]HYPERV-CANARY/Ignition/process/p_auxiliary/chlorine/pressure_transmitters/post_chlorine/totalizer'

ds = system.tag.queryTagCalculations(
    paths=[path],
    calculations=['shared.myMin', 'shared.myMax'],
    startDate=start,
    endDate=end,
    ignoreBadQuality=True)

for row in system.dataset.toPyDataSet(ds):
    print row[0], 'min:', row[1], 'max:', row[2]

Why read by index: a custom calculation has no registered name, so its column header is AD-HOC-PYTHON. Two custom calculations can both carry that header. The aliases argument renames the tag-path column (a working built-in call shows the alias in the tagpath column), not the calculation columns. Column order follows the order of calculations.

For a running totalizer, consumption over the range is max − min of the nonzero readings. This holds only if the totalizer did not reset or roll over inside the range; check the raw trend for a step down before trusting the difference.

Fallback: how do you compute the same result in script from raw history?

If the aggregate path is blocked, pull the stored rows and reduce them in the calling script. This moves the work from the gateway aggregate engine to the client, which is fine for one tag over weeks. It scales poorly across many tags or long ranges, because every raw row crosses the wire.

Avoid the string trap. set(ds.getColumnAsString(1)) - set([0]) removes nothing: the set holds strings such as "0.0", and the integer 0 never matches them. min() and max() over strings then compare lexicographically, so "1000" sorts below "999". Convert to float before filtering and comparing, as above.

How do you verify the fix end to end?

1. Confirm transport: rerun the call from the Script Console. No GatewayException and no NumberFormatException means hop 6 now parses.
2. Confirm typing: print [ds.getColumnType(i) for i in range(ds.columnCount)]. The two calculation columns must report a floating-point type, not Long.
3. Confirm shape: print ds.getColumnNames() shows tagpath plus two calculation columns. Swap the order in calculations and confirm the min and max columns swap with it.
4. Confirm the zero filter: pick a window that contains a recorded zero (find it in the raw pull) and query only that window. The minimum must equal the smallest nonzero stored reading, not 0 and not an interpolated in-between value.
5. Confirm against ground truth: run the raw-history fallback over the same range and compare. The aggregate min and max must match the script-side min and max of the nonzero stored values exactly.
6. Confirm window semantics: rerun with a returnSize greater than 1 and check each row against a raw pull of that window alone. Rows that only ever decrease (min) or increase (max) mean state is leaking through queryContext; return to Check 6.
7. Check the gateway log for entries from the shared.historyAgg logger and for any library import errors, then drop the logger back to its normal level.

FAQ