Troubleshooting S7-400 Profibus DP Station Failure 16#38C4 Errors

Problem Summary

An existing S7-400 installation with a CPU 414-2DP and two ET200M nodes on a single PROFIBUS DP master system exhibits an intermittent station failure on DP slave 4 (logical base input address 2040, DP master system ID 1). Two related events appear in the diagnostic buffer:

• Event ID 16#38C4 — Distributed I/Os: station return / station failure, calling OB86 (Rack failure OB) at priority class 26.
• Event ID 16#39B1 — I/O access error when updating the process image input table, P area, word access, access address 36, calling OB85 (Program execution error OB) at priority class 26.

The CPU Bus Fault (BF) LED is lit and the node-4 station LED is ON. Power cycling does not clear the fault, but the system self-recovers after several hours (4–5 h, not consistent). The same network on node 3 is unaffected. The failure occurs both on cold start after a long shutdown (e.g. weekend/holiday) and during steady-state operation. Power supplies, PROFIBUS cables, and bus connectors have already been replaced without resolution.

The 1994 timestamp on the diagnostic buffer entries is itself a tell-tale: it indicates the CPU clock has not been re-initialized since the factory or the buffer was carried over from initial commissioning. Without a valid clock, the time correlation between the 16#38C4 event and any field disturbance is impossible.

Event ID 16#38C4 — Distributed I/O Station Return

Event ID 16#38C4 is a DP diagnostic event in the S7-400/CPU 41x family. The complete diagnostic coding in the buffer for the observed entry is 16#38C4 1A56 C054 076C 07F8 0104. The fields are decoded as follows:

In the reported case the entry is logged as "Outgoing event" with the timestamp 11:51:46.553 PM 01/04/1994 — which is the "going-away" moment of slave 4. The fact that the bus fault LED is latched on and the station LED on node 4 is ON indicates the slave has remained physically unreachable for an extended period, even though OB86 has fired.

OB86 (Rack failure OB) is the only OB automatically started by the CPU when a DP slave fails or returns. If OB86 is not loaded, the CPU goes to STOP on the next failure event. Generating and downloading an OB86 with at least a TEMP variable declaration and a return is the first corrective action if not already present. Refer to the Siemens Online Support S7-400 manual set for the full OB86 local-data layout (OB86_EV_CLASS, OB86_FLT_ID, OB86_MDL_ADDR, OB86_Z23, OB86_Z24).

Event ID 16#39B1 — I/O Access Error on Process Image Update

Event ID 16#39B1 is generated when the CPU attempts to update the process image (P area access) but the addressed I/O is no longer reachable. The complete coding is 16#39B1 1A55 2050 2942 0000 0024, decoded as:

Address 36 (decimal) is byte 36, which is word 18 (QW 18 / IW 18). Combined with the configured base address of slave 4 = 2040 (input), the missing input is consistent with PIW 2076 (2040 + 36) if the OB is performing a partial PI update on the failing module. OB85 fires whenever the OB1 cycle tries to read an I/O word whose source is currently missing, which has the same root cause as the OB86 — a physical-layer interruption on slave 4.

Like OB86, OB85 must be loaded in the CPU. If OB85 is missing, the CPU goes to STOP on the first access to the failed area. The Siemens standard recommendation is to load OB85 and at minimum execute a clear of the failing input word inside it (e.g. L 0; T IW 2076;) to prevent re-entry on every OB1 scan.

Affected Hardware and Configuration

Configuration reported:

• CPU 414-2DP — order number 6ES7414-2XK05-0AB0 (or earlier -2XK04 / -2XK03). The integrated DP interface (X2) acts as the DP master for the network.
• Two ET200M nodes on PROFIBUS DP, addresses 3 and 4.
• ET200M interface modules: typically IM 153-1 (6ES7153-1AA03-0XB0) for standard operation, or IM 153-2 (6ES7153-2BA02-0XB0 / -2BA10-0XB0 / -2BA70-0XB0) for time-stamped / isochronous operation. IM 153-2 supports module-channel-level diagnostics.

Master system diagnostics address is 1900; slave 4 base input address is 2040. This implies the user has allocated input ranges starting at IB 2040 (PIW 2040..) for the modules under slave 4. Slave 3 occupies the range immediately below 2040; the BF only lights for slave 4, indicating the master itself is functional but cannot poll node 4 specifically.

Root Cause Analysis

The combination of (a) only slave 4 affected, (b) 4–5 h self-recovery, (c) failure to clear on power cycle, and (d) cold-start post-holiday repeatability points to one of three root-cause families, in order of probability:

1. Equipotential-bonding / ground loop on the PROFIBUS shield path. When the cold machine warms up, humidity and temperature drive leakage currents through the bus shield. A ground potential difference > ±7 V between two grounding points causes the RS-485 driver outputs of an ET200M IM to enter a constant fault state. This matches the 4–5 h thermal/condensation cycle and the post-holiday cold-start behavior.
2. IM 153-x interface module degradation on slave 4. Capacitor or opto-coupler degradation on the IM module itself can cause intermittent DP loss. A swap of node 3 and node 4 (with re-addressing via the DIL switch on the IM) is the definitive bench test.
3. Bus termination or stub-line reflection on the segment driving slave 4. A missing or doubled termination (only one resistor network should be ON per segment, located at the two physical ends), or a damaged T-connector, can cause slave 4 to lose synchronization while slaves 1, 2, 3 continue to communicate.

The fact that node 3 on the same cable run is healthy and that the BF remains latched for hours strongly suggests a hardware-layer or bonding issue, not a master-side configuration error. Configuration errors produce a hard, reproducible failure, not a 4–5 h thermal cycle.

Troubleshooting Matrix

Step-by-Step Diagnostic Procedure

1. Read the diagnostic buffer in STEP 7 (V5.x) or TIA Portal via Online > Online & Diagnostics > Diagnostic Buffer. Export the buffer (XML) for the entire timeframe covering 10 events before and 10 after the failure. Confirm the failure pattern is consistent and not a single transient.
2. Set the CPU clock (PLC > Set Time of Day) and enable Synchronize via NTP/SNTP if a route exists. Without a valid clock you cannot correlate PROFIBUS faults to field disturbances.
3. Load OB86 and OB85. If either OB is missing, generate it (right-click Block folder > Insert New Object > Organization Block > OB86 / OB85). At minimum the OB must contain BEGIN/END for OB86, and a default branch for OB85. This prevents STOP transitions on the next event.
4. Activate CPU diagnostic alarms: CPU Properties > Diagnostics/Clock > Report System Errors. This will auto-generate diagnostic DBs (e.g. DB125) containing event history for OB70–OB87, OB121, OB122. Refer to the Siemens TIA documentation on diagnostics blocks for reporting system errors (S7-300, S7-400) for the full block list and structure.
5. Read DP slave diagnostic data with SFC13 DPNRM_DG. The slave's full diagnostic telegram (6 bytes standard + module status) reveals whether the failure is "slave not reachable" (no response) versus "slave diagnostic: module failure / channel fault". The two cases indicate physical layer vs. module-side problems.
6. Install a PROFIBUS Diagnostic Repeater (6ES7972-0AB01-0XA0) between master and slave 4. The diagnostic repeater continuously monitors segment topology and broadcasts the exact failing segment/station to the diagnostic buffer.
7. Capture a PROFIBUS trace with a PROFIBUS analyzer (e.g. Softing PROFINET/PROFIBUS Diagnostics Suite, InPro S7) at the bus segment serving node 4. Look for SAP-lookup failures, token re-issues, and the exact byte time when node 4 stops responding.
8. Swap the IM 153 module on node 4 with a known-good spare. Re-address the new IM to address 4 (DIL switch 1=on, 2=on, 3=on, 4=on) and observe. If the failure moves to a different slave address, the IM is the cause.
9. Measure shield currents and ground potentials at every grounding clamp along the cable run. Use a clamp ammeter (Fluke 360 or equivalent) on the shield; currents above 100 mA indicate active ground loops. Measure between grounding points with a voltmeter; differences > 1 V DC or > 3 V AC require equipotential bonding.
10. Verify termination. A PROFIBUS DP segment must have exactly two termination resistors ON, one at each physical end, with both ON switches on the bus connector engaged. Mid-segment connectors must be set to OFF. Termination resistance is 220 Ω, pulled to +5 V via 390 Ω on each line pair (A and B), per IEC 61158-2.

Software Diagnostic Block Implementation

Use SFC13 DPNRM_DG to read the full diagnostic record of slave 4 into a data block on each OB86 call. In OB86, the local data OB86_EV_CLASS, OB86_MDL_ADDR, and OB86_Z23/OB86_Z24 give the slave address and event state. Save these into a ring buffer DB for post-mortem analysis.

Sample code in Structured Text (STEP 7 V5.5 / TIA Portal) for OB86:

// OB86 — Rack failure / station return
// OB86_EV_CLASS = B#16#38 (outgoing = fail) or B#16#39 (incoming = return)
// OB86_MDL_ADDR = configured diagnostic address of the slave

IF OB86_EV_CLASS = B#16#38 THEN
   "DiagDB".Failure[ "DiagDB".Idx ].SlvAddr := OB86_MDL_ADDR;
   "DiagDB".Failure[ "DiagDB".Idx ].State   := 0;  // 0 = failed
   "DiagDB".Idx := ( "DiagDB".Idx + 1 ) MOD 100;
END_IF;

IF OB86_EV_CLASS = B#16#39 THEN
   "DiagDB".Recovery[ "DiagDB".IdxR ].SlvAddr := OB86_MDL_ADDR;
   "DiagDB".Recovery[ "DiagDB".IdxR ].State   := 1;  // 1 = returned
   "DiagDB".IdxR := ( "DiagDB".IdxR + 1 ) MOD 100;
END_IF;

For OB85, the failing P-area word at byte 36 must be cleared to prevent OB85 from re-triggering on every scan while slave 4 is down. Sample:

// OB85 — I/O access error
// OB85_RESERVED_2 contains the failing I/O byte address (36)

IF OB85_RESERVED_2 = 36 THEN
   L 0;
   T IW 2076;   // default: replace failed input with 0
END_IF;

For a more detailed system diagnostic, enable Report System Errors (RSE) in HW Config. RSE auto-generates a set of FBs and DBs (notably DB 125 for event history) that are documented in the Siemens TIA documentation on diagnostics blocks for reporting system errors (S7-300, S7-400), including the call sequence and parameter assignment.

Hardware Inspection and Physical Layer Verification

Use a multimeter to verify the following at every connector on the segment serving node 4:

PROFIBUS DP cable spec (Siemens 6XV1830-0EH10, violet, sold per meter): characteristic impedance 135–165 Ω @ 3–20 MHz, loop resistance ≤ 110 Ω/km, capacitance ≤ 30 pF/m. Replacement cable must be ordered against this spec. Third-party substitutes that pass only DC continuity but fail the impedance spec are a frequent cause of intermittent DP errors on long stubs.

Connector (bus connector) part numbers:

• 6ES7972-0BA12-0XA0 — 90° outlet, no PG port
• 6ES7972-0BB12-0XA0 — 90° outlet, with PG port
• 6ES7972-0BA50-0XA0 — 35° outlet, no PG port (axial)

If the existing connectors were replaced during previous troubleshooting, confirm the PG-port version is not used in the middle of the segment — the PG port adds stub capacitance that can break termination.

Grounding, Shielding and Equipotential Bonding

Per the PROFIBUS Installation Guideline 8.012 (PNO) and IEC 61918, the bus shield must be:

1. Clamped to the grounding bar at both ends of the cable run using EMC cable glands that provide 360° contact (e.g. Weidmüller VEMC or Pflitsch).
2. Bonded to a single equipotential conductor (typically 16 mm² Cu) that runs parallel to and is bonded at both ends of the bus.
3. The shield must not be used as a protective-earth conductor (PE).

If the bus segments span multiple buildings, the equipotential bonding conductor must be 50 mm² Cu (or higher) and the bus must be isolated by OLM (Optical Link Module), e.g.:

• 6GK1502-3AB10 — OLM/G11 V4.0 (single fiber pair, BFOC)
• 6GK1503-2CB00 — OLM/G12-1300 (single mode, SC)

OLMs convert the electrical PROFIBUS segment to fiber optics, providing complete galvanic isolation and eliminating ground-loop currents. The trade-off is higher cost and the need to maintain the fiber (clean connectors, no sharp bends below 30 mm radius).

Inside a single building, ground loops are typically resolved by adding 16 mm² bonding jumpers between distribution cabinets, by separating the cabinet grounds from the building steel at one end, and by routing the PROFIBUS cable at least 200 mm from 400 V power cables and 1000 mm from VFD output cables.

Firmware and Configuration Verification

1. Check the CPU firmware version in PLC > Online & Diagnostics > Module Information. CPU 414-2DP variants 6ES7414-2XK05-0AB0 and -2XK04-0AB0 should be on firmware V7.0 or later; the XK05 supports V7.0.7 and later. See the Siemens firmware update support entry 109475012 for compatible firmware sets and the update procedure.
2. Check the IM 153 firmware for slave 4. The 6ES7153-2BA02 IM 153-2 supports firmware V8.x. Firmware V7.1 or later is recommended for diagnostic robustness on segments with marginal physical-layer quality.
3. Match the PROFIBUS baud rate to the segment length: 1.5 Mbit/s up to 200 m, 500 kbit/s up to 400 m, 187.5 kbit/s up to 1000 m. The reported baud rate is 1.5 Mbit/s; if the actual cable length to node 4 exceeds 200 m or includes a long stub, drop to 500 kbit/s to gain 12 dB of signal margin.
4. Confirm DP cycle and Ttr (target rotation time). Ttr must be ≥ 6 × Tbit × 11 + bus dead time. For 1.5 Mbit/s, Ttr ≥ 1500 × 11 × Tbit. An undersized Ttr will intermittently drop slaves 1 through n. With only 2 slaves, Ttr is not the cause; with 32 slaves it becomes relevant.
5. Regenerate the HW Config if any module under slave 4 has been added or replaced. The HW Config / GSD mismatch produces a hard failure, not an intermittent, so this is a sanity check.

Permanent Fixes: Topology, Isolation, Diagnostic Repeater

Recommended permanent measures, in order of robustness:

1. Add a PROFIBUS Diagnostic Repeater (6ES7972-0AB01-0XA0) at the DP master outlet. This device continuously monitors the segment and reports the exact location of the failure (segment number, station number) on the diagnostic buffer or via web server. It is the single most useful diagnostic tool on a marginal segment.
2. Insert an OLM (Optical Link Module) between master and slave 4, converting to fiber on the failing leg. This eliminates the ground-loop failure mode entirely. The master-side OLM replaces the direct bus connector; the slave-side OLM sits at node 4 entry.
3. Add a Repeater (6ES7972-0DA00-0AA0) on either side of node 4 to break the segment into two electrical sub-segments. Repeaters regenerate the RS-485 signal, providing a clean re-launch to node 4 and isolating the master from node 4's physical-layer disturbance.
4. Replace the IM 153 module on node 4 with a new (not refurbished) module. Verify the IM and the active bus module are seated correctly; bent pins on the bus module are a known issue on machines that have been disassembled for service.

Verification and Long-Term Monitoring

1. Set the CPU clock and re-arm the diagnostic buffer. With OB86/OB85 loaded, observe the buffer for one full production shift (or one full weekend, given the post-holiday pattern). Confirm that no new 16#38C4/16#39B1 events are recorded.
2. Compare against ground potential log: if a power-quality analyzer is available, log the PE–N voltage at the cabinet powering node 4 over the same period. Look for a correlation between ground-potential events and OB86 triggers.
3. Run a thermal cycle by switching off the cabinet climate control for 8 h, then powering up cold. A ground-loop / condensation issue reappears; a clean system does not.
4. Capture the diagnostic DB (e.g. DB 125 if RSE enabled) every 24 h via a script or HMI, and archive. This provides a forensic record if the fault returns in 6–12 months.
5. Document the resolution in the plant's maintenance log: root cause, fix applied, spare part replaced, and date. This step is non-technical but is the only way the next engineer finds the answer in 18 months.

FAQ