Fixing MindConnect Shared Secret Expiration on SIMATIC IOT2040

1. Problem Overview

The SIMATIC IOT2040 is a 32-bit Intel x86 IoT gateway designed for lightweight edge data acquisition in MindSphere / Industrial Edge environments. When the device is used with node-red-contrib-mindconnect and onboarded against a MindSphere Asset using the lightweight SHARED_SECRET profile, the agent's authentication key is bound to a strict validity window managed by the platform.

Operators have reported the following failure pattern in the field:

• The IOT2040 is switched off, powered down, or otherwise loses network reachability to the MindSphere tenant for approximately 15 consecutive days.
• On power-up, the gateway continues to run Node-RED, the MindConnect node reports a green or amber status, and the local flow keeps executing.
• No diagnostic surface inside Node-RED clearly indicates that the onboarded shared secret has been rotated or invalidated by the tenant.
• All time-series and event data captured during the offline window is silently dropped; values never appear in the corresponding asset's time series database.
• After the threshold is exceeded, the agent authentication fails permanently until a new onboarding is performed.

From the operator's perspective, the gateway appears healthy while the data is already lost. This article documents the root cause, the official recovery path, and a buffering pattern that closes the detection gap.

2. Affected Components and Versions

3. Root Cause Analysis

The onboarding flow used by the node-red-contrib-mindconnect package supports two key profiles, as documented in the package metadata:

• SHARED_SECRET – a 256-bit symmetric key for lightweight devices. This is the default profile on the IOT2040 and is generated and exchanged during the first onboarding handshake against the MindSphere agent onboarding endpoint.
• RSA_3072 – an asymmetric profile that requires the operator to provision a dedicated 3072-bit RSA key on the device before onboarding.

Once the 256-bit shared secret is provisioned, the IOT2040 stores it locally (typically in the MindConnect agent configuration directory used by the bundled MindConnect Library, e.g. /var/lib/mindconnect/ on the IOT2040 image) and uses it to authenticate every data upload, event, and bulk upload call against the MindSphere APIs.

On the tenant side, the onboarding service enforces a validity window on the issued client credential. When the gateway stays offline for an extended period, the tenant can rotate or invalidate the credential. After that point, the IOT2040's locally stored SHARED_SECRET is rejected on the next call.

The Siemens developer documentation states this explicitly:

"The client secret has expired you will have to onboard your agent again. This indicates that the agent authentication is not working anymore."

Source: MindConnect Node.js Troubleshooting.

Three operational realities compound the problem:

1. No local visibility. The node-red-contrib-mindconnect node exposes a status indicator, but it does not surface a distinct "credential expired" code. The flow user must inspect the underlying MindConnect Library log file to differentiate a network outage, a 401 with a fresh key, and a 401 with an expired shared secret.
2. No automatic buffering on the agent side. The default MindConnect configuration is store-and-forward on connect: as long as the local store buffer is not full, the agent holds data and forwards it when the next connection succeeds. If the local store fills up (default 16 MB on the IOT2040) before the device is re-onboarded, the oldest segments are discarded.
3. No silent re-onboarding. Re-onboarding requires a new sharedSecret to be issued by the tenant and written into the local configuration. There is no automatic rotation flow for SHARED_SECRET-based agents.

4. Symptoms and Diagnostic Procedure

Use the following check sequence to confirm that the device is in the "expired shared secret" state rather than a generic connectivity outage.

1. Open a shell on the IOT2040 (SSH on port 22, or console via the mini-USB service port) and tail the MindConnect Library log:

   tail -f /var/log/mindconnect/mindconnect.log

2. Trigger a manual upload from the Node-RED MindConnect "MindConnect Upload" node, or restart the MindConnect agent:

   systemctl restart mindconnect-agent

3. Inspect the response. A credential expiration surfaces as a 401 with payload fragments such as invalid_client, client_secret_expired, or expired in the JSON body returned by the token endpoint. A network outage produces a connection-refused or DNS error long before the authentication call is made.
4. Validate the local clock on the gateway:

   date
   timedatectl status

   A clock skew greater than a few minutes will produce a similar 401 even with a valid shared secret, and is a common co-cause.
5. Confirm the stored key file timestamp and size:

   ls -la /var/lib/mindconnect/
   wc -c /var/lib/mindconnect/sharedsecret.bin

   A 256-bit shared secret is exactly 32 bytes when stored raw. Any other size indicates file corruption or a non-default profile (e.g. RSA_3072 PEM block).

5. Official Recovery Procedure: Re-Onboard the Agent

The only officially supported recovery path is to re-onboard the agent. This regenerates the shared secret on the tenant and writes a fresh value to the device.

1. In the MindSphere / Insights Hub tenant, navigate to Asset Manager, select the affected IOT2040 asset, and remove the onboarded agent. If the asset is in a "rotten" state, use the Onboarding panel to revoke the existing agent credential.
2. On the IOT2040, stop the MindConnect service and any active Node-RED flows that write to the MindConnect node:

   systemctl stop mindconnect-agent
   systemctl stop node-red

3. Delete the locally cached credentials so the new onboarding handshake cannot be polluted by the old key:

   rm -f /var/lib/mindconnect/sharedsecret.bin
   rm -f /var/lib/mindconnect/agent.json
   rm -f /var/lib/mindconnect/config.json

4. Re-start the MindConnect onboarding helper. On the IOT2040 reference image, this is invoked through:

   /opt/mindconnect/mccl_onboard --profile SHARED_SECRET

   The onboarding helper prints a registration URL and a passphrase. Open the URL from a workstation, paste the passphrase, and assign the asset.
5. Verify the freshly stored key is exactly 32 bytes:

   wc -c /var/lib/mindconnect/sharedsecret.bin

6. Restart the agent and Node-RED:

   systemctl start mindconnect-agent
   systemctl start node-red

7. From Node-RED, perform a single test upload using the MindConnect "MindConnect Upload" node and confirm the response code is HTTP 202 and the asset time series shows the new sample within the tenant's UI refresh interval (typically 30-60 s).

After re-onboarding, the time-series database will have a gap for the offline window. The gap cannot be backfilled by the agent; the only way to reconstruct it is from an on-device buffer (see Section 6) or from a duplicate acquisition path (e.g. the PLC's own historian).

6. Closing the Data Gap: Local Buffering in Node-RED

To prevent the next outage from re-creating the same gap, buffer the acquired values on disk while the MindConnect node is unable to forward them, and replay the buffer after a successful upload.

6.1 Buffer design

• Use a flat-file JSON Lines store under /var/lib/node-red/buffer/ on the IOT2040's internal flash. Each line is one acquisition record with timestamp, value, and a hash of the upstream identity (asset, aspect, variable).
• Cap the buffer at a fixed size (for example 8 MB on the 8 GB IOT2040 SKU) and use a ring-file rotation: when the active file exceeds the cap, rename it to buffer-1.jsonl and start a new buffer-0.jsonl. Delete the oldest file when there are already two, so the buffer never exceeds 16 MB.
• Mark each record with an upload status of pending, uploaded, or dropped to make audits possible after a forced restart.

6.2 Node-RED flow skeleton (functional outline)

[PLC] --> [Read block] --> [Buffer write]
                                   |
                                   v
                          [Status: pending]
                                   |
              (every 60 s)        v
                          [Read pending batch]
                                   |
                                   v
                       [MindConnect Upload]
                                   |
                   +---------------+---------------+
                   |                               |
                success                          error
                   |                               |
                   v                               v
            [Mark uploaded]                [Keep pending, raise alarm]

6.3 Implementing the buffer write in a function node

// Function node: "Buffer Write"
const fs = global.get('fs') || require('fs');
const path = '/var/lib/node-red/buffer/buffer-0.jsonl';
const record = {
    ts: new Date().toISOString(),
    asset: msg.asset,
    aspect: msg.aspect,
    variable: msg.variable,
    value: msg.payload,
    quality: msg.quality || 'good',
    status: 'pending'
};
fs.appendFileSync(path, JSON.stringify(record) + '\n');
return msg;

6.4 Replay loop

// Function node: "Replay Pending"
const fs = require('fs');
const dir = '/var/lib/node-red/buffer/';
const out = [];
fs.readdirSync(dir)
  .filter(f => /^buffer-\d+\.jsonl$/.test(f))
  .forEach(f => {
      const lines = fs.readFileSync(dir + f, 'utf8').split('\n').filter(Boolean);
      for (const line of lines) {
          const r = JSON.parse(line);
          if (r.status === 'pending') {
              out.push({
                  asset: r.asset,
                  aspect: r.aspect,
                  variable: r.variable,
                  payload: r.value,
                  quality: r.quality
              });
          }
      }
  });
return [out];

6.5 Marking successful uploads

// Function node: "Mark Uploaded" (input: MindConnect Upload response)
const fs = require('fs');
const dir = '/var/lib/node-red/buffer/';
if (msg.statusCode === 202) {
    fs.readdirSync(dir)
      .filter(f => /^buffer-\d+\.jsonl$/.test(f))
      .forEach(f => {
          const content = fs.readFileSync(dir + f, 'utf8');
          const updated = content
              .split('\n')
              .filter(Boolean)
              .map(line => {
                  const r = JSON.parse(line);
                  if (r.status === 'pending') r.status = 'uploaded';
                  return JSON.stringify(r);
              })
              .join('\n');
          fs.writeFileSync(dir + f, updated);
      });
    node.status({ fill: 'green', shape: 'dot', text: 'buffer flushed' });
} else {
    node.status({ fill: 'red', shape: 'ring', text: 'auth or network error' });
}
return msg;

7. Detection: Raising an Alarm on Credential Expiry

Detection is the missing piece in the default flow. A practical way to make the expiration visible is to inspect the MindConnect Library log file for the substring client_secret combined with expired or invalid_client, and to push that as an alarm into Node-RED itself or via SMTP.

// Function node: "Watch MindConnect Log"
const fs = require('fs');
const logfile = '/var/log/mindconnect/mindconnect.log';
let tail = context.get('tail') || '';
const stat = fs.statSync(logfile);
if (stat.size < (context.get('size') || 0)) tail = ''; // rotated
context.set('size', stat.size);
const fd = fs.openSync(logfile, 'r');
const buf = Buffer.alloc(stat.size - tail.length);
fs.readSync(fd, buf, 0, buf.length, tail.length);
fs.closeSync(fd);
tail += buf.toString('utf8');
context.set('tail', tail);

const hit = /client_secret[^
]*expired|invalid_client/i.test(tail);
if (hit && !context.get('alarmActive')) {
    context.set('alarmActive', true);
    node.warn('MindConnect shared secret has expired. Re-onboard required.');
    msg.payload = 'MindConnect shared secret expired on IOT2040';
    msg.topic   = 'mindconnect/auth';
    return [null, msg]; // second output: alarm
}
if (!hit && context.get('alarmActive')) context.set('alarmActive', false);
return [msg, null];

Wire the alarm output to an e-mail node, an MQTT publish node, or a write to a dedicated MindSphere "diagnostic" aspect. The alarm gives operations a hard signal that re-onboarding is required before the next buffer rotation drops the offline data.

8. Prevention

9. Verification and Sign-off Checklist

1. Confirm a successful onboarding response is logged with HTTP 200 / 202 in /var/log/mindconnect/mindconnect.log.
2. Confirm the sharedsecret.bin file on the IOT2040 is 32 bytes (256 bits) for SHARED_SECRET, or a valid PEM block for RSA_3072.
3. Trigger a forced outage (pull the WAN cable) and confirm that buffered values are still replayed when the connection is restored.
4. Trigger a forced offline window long enough to cross the tenant's expiration threshold and confirm that the alarm node fires with a clear message, not just an amber MindConnect node status.
5. After re-onboarding, confirm the new onboarding is reflected in the tenant's Asset Manager > Onboarding view and that the new sharedSecret is bound to the correct asset and aspect.
6. Verify that the time-series database shows a contiguous timeline from the moment the buffer was flushed onward, with a single, documented gap for the pre-re-onboarding window.

10. Troubleshooting Matrix

11. Field-Proven Caveats

• The IOT2040 reference image ships Node-RED 0.20.x with an older Node.js runtime. Some buffer functions that use fs.promises or modern Array methods must be transpiled or guarded with feature checks before deployment.
• The 256-bit SHARED_SECRET is a raw key, not a JWT. Do not paste it into a Mindsphere Authorization: Bearer header. The MindConnect Library performs the OAuth2 client-credentials exchange for you.
• If multiple Node-RED instances are started (e.g. node-red started manually and again via systemd), they will fight for the same sharedsecret.bin and produce spurious 401s that look exactly like an expired key. Check pgrep -af node-red first.
• The "15 days" symptom reported in the field is a tenant-side value, not a fixed product number. Different MindSphere releases and Insights Hub tenants may use different validity windows. Treat the number as a guideline and configure your buffer to cover the worst-case window you have observed in your environment.

12. Frequently Asked Questions