Resolving ClamAV High RAM Usage on Siemens IOT2050 Industrial OS

Resolving ClamAV High RAM Usage on Siemens IOT2050 Industrial OS

The Siemens SIMATIC IOT2050 is a compact industrial IoT gateway built around an ARM Cortex-A53 quad-core SoC with either 1 GB or 2 GB of DDR3 RAM and a 4 GB eMMC flash. When it ships with Siemens Industrial OS, the Debian-based image used for SCADA edge, OPC UA bridging, and I/O aggregation, several packages are preinstalled to support runtime services. One of these, ClamAV (Cisco Talos' open-source antivirus engine), has been reported to grow its resident memory footprint without an upper bound, eventually saturating the gateway's 1 GB variant and degrading any control-plane service that depends on the same memory pool (Node-RED, S7 communication, the iot2050-controller service, etc.).

This reference documents the root cause of the leak, the safe purge procedure, the configuration tuning that prevents recurrence when ClamAV must remain installed (for regulatory or audit reasons), and the verification commands that prove the fix.

1. Affected Hardware and Software Versions

Always confirm the installed version with the commands in Section 4 before applying any fix. If the field engineer is operating against an image signed by an internal release pipeline, capture the exact /etc/os-release output and the result of dpkg -l | grep -i clam for the change record.

2. Problem Summary

Operators report that the IOT2050 gateway becomes "super laggy" without warning. Running free -h shows buff/cache and used climbing steadily while available collapses toward zero. Running top, htop, or ps -o pid,user,vsz,rss,comm -C clamd reveals that clamd (the on-access scanning daemon) owns the bulk of the resident memory and is continuously allocating without releasing. Within 24 to 72 hours of boot, the daemon can hold 600 MB to 1.2 GB of RSS on a 1 GB gateway, which forces the kernel into a thrashing state and starves every other process.

3. Root Cause Analysis

Three independent factors combine to create the memory blow-up:

1. Unbounded cache growth. ClamAV's signature loader (the bytecode interpreter and signature matching engines) allocates its loaded signature trie, the Yara-like matching structures, and the in-memory pending file cache without enforcing a hard RSS ceiling. The maintainers track this behaviour publicly: GitHub issue #565 "too much RAM is being used" in the Cisco Talos ClamAV repository states that "you will always need at least 2 GB of RAM to run clamD" and that "3 GB or even 4 GB is required, particularly if you want concurrent database" updates.
2. Default Industrial OS configuration leaves ClamAV enabled. The image activates clamav-daemon.service and schedules freshclam for hourly signature pulls. There is no MemoryMax= or MemoryHigh= slice defined for clamav.service, so the daemon is free to consume the entire RAM pool.
3. Industrial workloads are non-target by design. Most IOT2050 deployments serve deterministic control traffic (PROFINET, OPC UA, MQTT to on-prem brokers). The data ClamAV inspects is structured (numeric, short telemetry strings) and is not a typical malware vector. Running a full antivirus workload on every byte the gateway handles offers a near-zero detection rate while consuming 100% of the available headroom.

4. Symptoms and Diagnostic Commands

Run the following diagnostics in order. Capture each output to the maintenance ticket:

1. Confirm RAM pressure:

   free -h
   cat /proc/meminfo | grep -E '^(MemTotal|MemAvailable|MemFree|Buffers|Cached|Slab):'

2. Identify the offending process:

   top -o %MEM -b -n 1 | head -20
   ps -eo pid,user,pri,vsz,rss,pcpu,pmem,args --sort -rss | head -15
   pgrep -a clamd

3. Confirm the package version:

   dpkg -l | grep -iE 'clam|libclam'
   clamd --version
   cat /etc/os-release

4. Check whether the on-access scanner is armed:

   systemctl status clamav-daemon.service clamav-freshclam.service
   cat /etc/clamav/clamd.conf | grep -E '^(OnAccess|MaxFileSize|MaxRecursion|StreamMaxLength)'
   ls -la /var/lib/clamav/

5. Confirm the industrial OS release:

   cat /etc/iot2050-release 2>/dev/null || true
   dpkg-query -W iot2050-firmware iot2050-controller industrial-os-release 2>/dev/null

If clamd holds the largest RSS line and is growing monotonically across two samples taken 10 minutes apart, you have the leak.

5. Prerequisites for Removal

• Console or SSH access to the IOT2050 with root or sudo rights.
• Out-of-band recovery path: Industrial OS recovery image on USB stick in case the purge corrupts the running system. The Industrial OS recovery ISO is delivered through the Siemens Industry Online Support portal entry SIMATIC IOT2050 product support.
• Schedule a maintenance window of at least 10 minutes of downtime; the post-install image rebuild can take longer on slow eMMC.
• If ClamAV is part of a regulated security baseline (e.g., IEC 62443 zone enforcement), escalate before removal and have a written change window.

6. Step-by-Step Removal

Use the aggressive purge sequence. The plain apt-get remove will leave configuration directories and the clamav user behind because the package ships with a retention hook that resists partial uninstall. The verified working sequence is:

sudo apt-get update
sudo systemctl stop clamav-daemon.service clamav-freshclam.service
sudo systemctl disable clamav-daemon.service clamav-freshclam.service
sudo apt-get remove --purge -y clamav clamav-base clamav-daemon clamav-freshclam \
     libclamav9 clamav-milter
sudo apt-get autoremove --purge -y
sudo deluser --remove-home clamav 2>/dev/null || true
sudo delgroup clamav 2>/dev/null || true
sudo rm -rf /var/lib/clamav /etc/clamav /var/log/clamav /run/clamav
sudo systemctl daemon-reload
sudo systemctl reset-failed

Verify the removal:

dpkg -l | grep -i clam
which clamd clamdscan clamscan freshclam
id clamav
ls -la /var/lib/clamav 2>stdout
systemctl list-unit-files | grep -i clam

All four greps must return empty. If dpkg -l still lists any clamav-* package, repeat the purge or run sudo dpkg --purge <package> for each remaining entry. This is the path operators have used on Industrial OS V2.1.1 to reach a clean state without reflashing.

7. Keeping ClamAV: Capping Memory in clamd.conf

If security policy requires the antivirus to remain, the daemon must be placed under a hard memory ceiling. Two layers are needed: a Linux cgroup slice limit enforced by systemd, and an internal ClamAV scan-size guard. Apply both.

7.1 Systemd memory slice

Create a drop-in override:

sudo systemctl edit clamav-daemon.service

Paste the following into the editor:

[Service]
MemoryMax=384M
MemoryHigh=320M
OOMScoreAdjust=500
MemorySwapMax=0
Restart=on-failure
RestartSec=30

Activate with:

sudo systemctl daemon-reload
sudo systemctl restart clamav-daemon.service
systemctl show clamav-daemon.service | grep -E 'Memory(High|Max)|OOM'

7.2 clamd.conf internal ceilings

Edit /etc/clamav/clamd.conf and set sane I/O limits so that the scan engine does not need to buffer large files in memory. The relevant directives (see the ClamAV configuration reference) are:

Reload and validate:

sudo systemctl restart clamav-daemon.service
sudo clamd --debug 2>&1 | grep -iE 'config|maxfilesize|maxrecursion'
sudo clamdscan --infected --no-summary /var/log/iot2050/

8. Mitigation Without Removal: Stopping the On-Access Path

If the operations team wants to keep the package but stop the leak, disable the on-access scan and let clamscan run as a scheduled, low-priority batch job (for example, weekly at 03:00) instead. The leak is primarily driven by the always-on on-access hook: the daemon never releases the loaded signature cache while it is armed.

sudo systemctl stop clamav-daemon.service
sudo systemctl disable clamav-daemon.service
sudo systemctl mask clamav-daemon.service
sudo systemctl enable --now clamav-freshclam.timer
sudo clamscan -ri /var/lib/iot2050 /srv/edge --log=/var/log/clamav/weekly.log

Schedule the second line through crontab -e:

0 3 * * 0 /usr/bin/nice -n 19 /usr/bin/clamscan -ri /var/lib/iot2050 /srv/edge --log=/var/log/clamav/weekly.log
30 3 * * 0 /usr/bin/freshclam --quiet

Weekly scans hit the eMMC once and exit, returning RSS to the kernel. Memory pressure during scan will peak at < 200 MB with the limits from Section 7.2.

9. Verification Procedure

1. After removal or capping, sample RSS over time. Run twice, separated by 5 minutes, and confirm the value does not grow:

   for i in 1 2; do date; ps -o pid,rss,vsz,comm -C clamd 2>/dev/null; sleep 300; done

2. Confirm RAM pressure has cleared:

   free -h; cat /proc/meminfo | grep -E '^(MemAvailable|Slab):'

3. Confirm the daemon obeys the cgroup slice (if kept):

   systemd-cgtop -n 1 -d 2 | grep -E 'clamav|/'
   systemctl show clamav-daemon.service -p MemoryCurrent,MemoryHigh,MemoryMax

4. Confirm the gateway is responsive:

   curl -fsS http://127.0.0.1:1880/redact  # Node-RED, if installed
   ping -c 5 192.168.0.1
   ps -o pid,comm,etime -p $(pidof iot2050-controller)

5. Confirm signature freshness (if keeping ClamAV):

   freshclam -V
   sigtool --list-sigs | wc -l

If clamd is not present, the relevant verification is that pgrep -a clamd returns empty and no clamav-* package is listed in dpkg -l. The Industrial OS control plane (Node-RED, iot2050-controller, the IOT2050 web configurator) will reclaim the previously occupied RSS in seconds, and the gateway returns to its normal latency profile.

10. Risk and Security Considerations

Removing ClamAV trades one availability problem for one potential confidentiality gap. Evaluate before acting:

• Threat model. If the IOT2050 only ingests data from trusted controllers (S7-1200 / S7-1500, OPC UA server with mutual TLS), the malware injection surface is near zero. freshclam signatures provide no extra protection against a misconfigured TIA Portal project.
• False positives. Compiling firmware updates, OPC UA namespace files, or Siemens safety signatures through ClamAV has caused on-access quarantine of legitimate binaries in production. The availability loss outweighs the integrity gain.
• Update hygiene. If ClamAV is required to stay, set freshclam to check at most once per day; the default hourly cadence adds load without benefit and can re-enter the unbounded cache path.
• Network exposure. When the IOT2050 sits in a DMZ and accepts files from outside (SMB shares, USB mounts, REST uploads from cloud), keep the antivirus but with the memory caps in Section 7.

11. Troubleshooting Matrix

12. Re-enabling ClamAV Cleanly

If policy later reverses and ClamAV must come back, install the package set fresh, apply the systemd drop-in from Section 7.1, and tune clamd.conf in the same pass. Do not restore the original clamd.conf from backup; re-author it with the limits in Section 7.2 so the leak cannot return.

sudo apt-get install -y clamav clamav-daemon
sudo systemctl edit clamav-daemon.service    # paste the MemoryMax= stanza
sudo freshclam                               # download current signatures first
sudo systemctl enable --now clamav-daemon.service
sudo systemctl status clamav-daemon.service -l

13. Field-Engineer Quick Reference

With the daemon removed or properly caged, the IOT2050 returns to a stable operational state and the available memory on the 1 GB variant climbs from a few hundred megabytes back to 700 MB or higher under typical edge workloads. Operators should still log a follow-up review at the next quarterly maintenance window to confirm the issue has not reappeared via a package update that re-enables clamav-daemon.service by default.