Troubleshooting Ignition Gateway Log Database Errors

Ignition Gateway serves the log page by querying a local SQLite database. Follow the request from the browser to route-group=status, through route-path=/logs, into GatewayLoggingManagerImpl and SQLiteAppenderReader, and finally to system_logs.idb. When database maintenance holds the file, another process retains a connection, or the database is damaged, the query stops at the SQLite layer and the log list can alternate between populated and empty states.

Where does the log request travel and stop?

Layer one first. The relevant physical path is the gateway host's storage: on the reported Raspberry Pi 4 installation, that means the filesystem and its underlying storage. Confirm that the filesystem accepts persistent writes before treating the failure as a web or SQL problem. A read-only or overlay-backed filesystem can discard a deletion at reboot and reveal the previous files again.

The displayed query error is therefore not, by itself, proof of database corruption. The stack trace and the nested SQLite exception decide whether the immediate cause is contention, query construction, or a damaged database. If the web page cannot display the details, read wrapper.log directly from the gateway filesystem.

Which symptoms distinguish cleanup, locking, and corruption?

Two installations provide useful version boundaries without defining a universal affected-version range: one reported the condition on 8.0.6; another saw it after moving from 8.1.19 to 8.1.22. Diagnose the exception and maintenance timing on the installed version rather than assigning every occurrence to an upgrade defect.

Which recovery approach fits the failure?

Start with maintenance correlation because the reported large database recovered after cleanup completed. Use configuration tuning if the condition repeats. Rebuild only the SQLite event database when the lock has cleared but queries still fail. Reinstalling the gateway is unnecessary for a recoverable log-database problem and does not correct an overlay filesystem that restores old data.

Why can normal log maintenance block the web page?

The SQLite appender stores events in one database file and periodically removes old rows. A maintenance operation and the web reader can contend for that file. One captured sequence reported Max entries: 50000 and Max filesize: 104857600, followed by Starting logfile maintenance and then an error on /logs. Cleanup was observed taking more than five seconds, long enough for a concurrent web request to fail.

Most stored events were described as occupying approximately 600–800 bytes on disk. That is an event-size estimate, not a database-capacity formula; indexes, pages, free space, and SQLite overhead also consume the file. A list containing about 260,000 entries can exceed the configured retention target because entryLimit is enforced during maintenance rather than synchronously on every insert. With size cleanup removing 500 old events per cycle, reduction is incremental.

How should the maintenance settings be changed?

The maintenance block in a fresh data/logback.xml is commented out. Activating it turns the displayed values into explicit configuration. Change one retention dimension at a time so the resulting database size, maintenance duration, and event history remain attributable to that change.

1. Copy the existing data/logback.xml before editing it.
2. Record the current database size, approximate event count, and timestamps of Scheduling logfile maintenance, Starting logfile maintenance, and the query failure.
3. Choose the controlling objective: use entryLimit to target an event count and maxDatabaseSize to cap database growth by bytes.
4. Set diskspaceCleanupEventCount according to the required recovery rate. Increasing it removes more old rows after a size breach but can lengthen an individual maintenance operation.
5. Use maxEventsPerMaintenance and minTimeBetweenMaintenance to control how often maintenance runs. The time limit takes precedence over the event-count trigger.
6. Restart or reload the gateway by the configuration-management method used for the installation, then watch wrapper.log through at least one maintenance cycle.

Do not point scripts or database utilities at system_logs.idb while the gateway is running. SQLite file locking protects transactional integrity, and an extra connection can produce SQLITE_BUSY even when retention settings are reasonable.

How can a damaged log database be rebuilt safely?

Rebuild the internal event database only after capturing the complete error from wrapper.log. Removing every file from the logs directory mixes separate logging systems and discards evidence needed to diagnose the failure.

1. Stop the Ignition Gateway so no writer or reader holds the database.
2. Preserve a copy of wrapper.log and the failed system_logs.idb if the exception must be investigated later.
3. Delete only system_logs.idb from the logs directory. On the reported Linux installation, the directory was /usr/local/ignition/logs/.
4. Start the gateway. The logging subsystem creates a new system_logs.idb.
5. Open the gateway log page and confirm that new events appear without a query exception.

After a clean deletion and startup, the expected initial files were a new system_logs.idb and a single wrapper.log until wrapper rollover occurs. The wrapper file is managed by the service wrapper, and its rollover configuration belongs in ignition.conf, not the SQLite maintenance block.

Why can deleted logs return after a reboot?

Ignition creates a new empty log database when its database file is absent; it does not repopulate that file with deleted historical events. If old events return, follow the storage path below the application. An overlay filesystem can present a read-only base image while holding changes in volatile storage. Rebooting discards the volatile layer and exposes the old base copy again. A restored disk image or backup can produce the same visible result.

1. Stop the gateway and verify that its process no longer has the log files open.
2. Delete system_logs.idb, then confirm the file is absent before restarting the gateway.
3. Create or modify a harmless test file outside the Ignition directory, reboot the host, and check whether that change persists.
4. Inspect the Raspberry Pi operating-system configuration for an overlay or read-only filesystem if both changes revert.
5. If the file reappears while the gateway remains stopped, inspect operating-system restore, imaging, and backup processes rather than the Ignition logging configuration.

How is the fix verified from end to end?

1. Monitor activity directly with tail --follow wrapper.log. This retains message line breaks and captures traffic that can bypass the database-backed log viewer.
2. Open the gateway log page and generate normal log activity without repeatedly refreshing during an active maintenance operation.
3. Confirm that /logs returns records and that neither Unable to build a valid query with the parameters provided nor [SQLITE_BUSY] appears.
4. Wait for the next scheduled maintenance cycle and measure the interval from its start to completion.
5. Confirm that the retained event count and system_logs.idb size move toward the configured limits.
6. When storage persistence was part of the fault, reboot the host once and verify that the newly created database remains current rather than reverting to deleted historical data.

FAQ