CPwE Parallel Redundancy Protocol Design Considerations

This chapter describes design considerations and configuration recommendations when implementing Parallel Redundancy Protocol (PRP) in an IACS architecture. This includes guidelines for creating redundant EtherNet/IP network topologies in a Cell/Area Zone using PRP, and connecting PRP topologies to a larger plant-wide or site-wide network using redundant distribution and RedBox switches.

Parallel Redundancy Protocol Overview

PRP is defined in the international standard IEC 62439-3 and provides high availability in Ethernet networks. PRP implements redundancy by using PRP-enabled nodes (IACS devices) that send duplicate Ethernet frames to two fail-independent network infrastructures, known as LAN A and LAN B.

PRP technology is well suited for a variety of critical infrastructure IACS in process and heavy industries that require continuous, high availability operation. Advantages of using PRP over other network resiliency technologies include:

  • No IACS data loss during a single fault in LAN A or LAN B
  • Protection against extended infrastructure failures in a single LAN (e.g., maintenance work, prolonged power outages, multiple network faults)
  • Recovery after multiple faults in certain situations depending on the LAN topologies
  • Flexibility of allowing various network topologies, resiliency protocols, and IES platforms for each LAN
  • Ease of migration from non-Ethernet redundant media technologies such as ControlNet ® Networks (not covered as part of CPwE PRP)

Important factors when planning to implement PRP technology are:

  • Support of PRP by IACS devices
  • Possibility of building two independent network topologies without common faults
  • Connectivity to non-PRP parts of the plant-wide or site-wide infrastructure
  • Configuration of other network services such as multicast management and time synchronization for optimal operation in the PRP network.

The following sections provide a brief overview of the PRP operation, components, and topologies. For more details refer to:

Parallel Redundancy Protocol Components

A PRP network includes the components shown in Table 2-1 .

 

Table 2-1 PRP Components

Component
Description
Examples

LAN A and LAN B

Redundant, active Ethernet networks that operate in parallel and are fault independent.

Linear, star, redundant star or ring topology using managed switches

Double attached node (DAN)

An IACS device with PRP technology that connects to both LAN A and LAN B.

1756-EN4TR ControlLogix Ether-Net/IP module (firmware 4.001 or later)

1756-EN2TP ControlLogix Ether-Net/IP module

5094-AENTR Flex 5000™ Ether-Net/IP module

Single attached node (SAN)

An IACS device without PRP technology that connects to either LAN A or LAN B. A SAN does not have PRP redundancy and typically is a non-critical device or its function is duplicated in both LANs.

An HMI terminal, a thin client, an industrial PC

 

Redundancy box (RedBox)

An IES with PRP technology that connects non-PRP IACS devices or non-PRP part of the network to both LAN A and LAN B.

Stratix 5400, Stratix 5410 and Stratix 5800 managed switches

Virtual double attached node (VDAN)

An IACS device without PRP technology that connects to both LAN A and LAN B through a RedBox. A VDAN has PRP redundancy and appears to other nodes in the network as a DAN.

Drives, I/O, controllers without PRP support

Infrastructure switches

Switches in LAN A or LAN B that are not configured as a RedBox.

Stratix managed switches

Parallel Redundancy Protocol Operation

An IACS device with PRP technology (a DAN) has two Ethernet ports that operate in parallel and attach to independent LAN A and LAN B. During normal network operation, a DAN simultaneously sends and receives duplicate Ethernet frames through both ports.

The receiving node accepts whichever frame arrives first and discards the subsequent copy. If a failure occurs in one of the LANs, traffic continues to flow through the other LAN uninterrupted with no convergence time.

Unlike other resiliency protocols, such as Spanning Tree Protocol (STP) or DLR, PRP does not require reconfiguration in LAN A or B after the fault (e.g., unblocking the port). PRP provides redundancy by using duplicate network infrastructure rather than redundant paths in the same network.

Figure 2-1 Redundant Path versus Redundant Networks

 

387864.jpg

DAN Operation

A DAN has two Ethernet ports that are attached to the upper communication layers of the IACS device through the Link Redundancy Entity (LRE). The LRE handles duplication of packets and management of redundancy (Figure 2-2). The upper layers are unaware of redundancy because the LRE provides to them the same interface as a non-redundant network adapter.

note.gif

Noteblank.gif A DAN uses the same MAC address and IP address to communicate on both LANs.


Figure 2-2 PRP DAN Communication Layers

 

387865.jpg

When a DAN sends a frame to another DAN:

  • The LRE creates two copies of the frame and sends them through LAN A and LAN B ports with a Redundancy Check Trailer (RCT) appended to each frame. The 6-byte trailer contains a sequence number, the LAN identifier, frame data size, and the PRP suffix that identifies the trailer type as PRP (Figure 2-3).
  • The duplicate frames traverse the two LANs, perhaps under different network conditions and with slightly different delays, and arrive at the destination node.
  • The LRE in the destination DAN forwards the first received copy of the frame to the upper layers (without the PRP trailer) and discards the second copy (if it arrives).
  • PRP algorithm is designed in a way that it should never reject a legitimate frame, however in rare cases a duplicate frame can be accepted as a new one and passed to the upper layers. This could happen if the duplicate frame arrives with significant time difference. Upper layer protocols (TCP or EtherNet/IP) are able to handle occasional duplicate frames.

Figure 2-3 Ethernet Frame with RCT Appended

 

387866.jpg
note.gif

Noteblank.gif The RCT trailer adds six bytes to an Ethernet frame. To accommodate a maximum size Ethernet frame (1500 bytes) with the PRP trailer attached, all LAN A and LAN B network devices should be configured with the maximum transmission unit (MTU) size of at least 1506 bytes. This is not required for a RedBox IES.


RedBox and VDAN Operation

The RedBox device acts as LRE for one or several connected VDANs or for a non-PRP bridged network segment. The RedBox keeps track of sequence numbers and handles duplicate received frames for multiple VDANs.

Two Gigabit Ethernet ports on the RedBox IES are configured as one logical interface—a PRP channel group (Figure 2-4). The PRP ports can be in access mode for single VLAN deployments, trunk mode to support multiple VLANs, or routed mode. In the channel group, the lower numbered member port is the primary port and connects to LAN A. The higher numbered port is the secondary port and connects to LAN B.

Figure 2-4 RedBox PRP Channel

 

387867.jpg
  • The Stratix 5400 IES supports one PRP channel. The Stratix 5410 and Stratix 5800 IES support up to two PRP channels.
  • Only certain pairs of ports can be used in a PRP channel, depending on the platform.
  • A maximum of 512 VDAN entries are supported in the PRP VDAN table. If the VDAN table is full, the switch cannot send supervisory frames for new VDANs.
  • The RedBox IES supports a maximum of 512 SAN and DAN entries in the Node table.
  • Ports in the PRP channel group cannot be configured for other resiliency protocol, e.g., DLR or Resilient Ethernet Protocol (REP).
  • Once the PRP ports are added to the group, individual port settings should not be changed unless the port is removed from the group.

In addition to connecting VDANs, a RedBox IES in the PRP network is necessary in following situations:

  • Routing is enabled in the network.
  • Connectivity to a non-PRP LAN is required, e.g., a DLR segment, a plant-wide or site-wide connectivity.
  • Internet Group Management Protocol (IGMP) querier role is required for multicast management.
  • Boundary clock role is required for plant-wide or site-wide time distribution using PTP.

Recommendations for configuring RedBox IES for these use cases are described in later sections of this Design and Implementation Guide.

For more information about Stratix switch PRP functionality and configuration, refer to:

note.gif

Noteblank.gif A RedBox IES in a PRP system is a single point of failure. IACS availability requirements should be evaluated when connecting critical devices to a RedBox. Best network practices must be implemented, such as using redundant power supplies, installing proper cabling and grounding, and avoiding uncontrolled loops in the LAN A and LAN B topologies.


SAN Operation

Devices without PRP support (SAN) can be included in the PRP topology as non-redundant devices connected to either LAN A or LAN B:

  • A SAN can accept and process Ethernet frames with the RCT attached. The SAN simply ignores the PRP trailer as the Ethernet padding in the frame.
  • To avoid duplication of packets for SANs, the DAN or the RedBox IES keeps track of learned MAC addresses in the PRP node table, identifies the device as attached to only one LAN, then sends the frame to that LAN only without the PRP trailer.
  • The SAN traffic from one of the LANs can be received and processed by the destination DAN in a normal way.
note.gif

Noteblank.gif All SANs must have unique IP addresses across the PRP network. Address Conflict Detection (ACD) mechanisms may fail if duplicate addresses are assigned to SANs in different LANs. To avoid that, use RedBoxes and connect devices as VDANs.


Network Management and Supervision

Benefits of network redundancy can only be realized if the network status and performance is monitored. This can be achieved with a Network Monitoring Tool (NMT) using Simple Network Management Protocol (SNMP) and other management protocols, EtherNet/IP diagnostic tools, and diagnostic information available in PRP-capable IACS devices and RedBox IES.

PRP nodes (DAN or RedBox) provide real-time PRP statistics and verify if frames from known DANs or VDANs are received via both PRP ports. For more information, refer to Chapter4, “CPwE Parallel Redundancy Protocol Monitoring and Troubleshooting”

Each DAN periodically sends a PRP supervision frame that announces its presence on the network and allows other nodes to check the health of the PRP network. The RedBox sends supervisory frames on behalf of connected VDANs. The supervisory frames are Layer 2 multicast Ethernet frames sent to a reserved multicast MAC address.

note.gif

Noteblank.gif An NMT should be connected to the PRP network via a RedBox to access IACS devices and IES in both LANs. While LAN A and LAN B are isolated on the network layer, all managed IES and IACS devices should have different IP addresses within and between each LAN for management purposes.


Parallel Redundancy Protocol Network Design Recommendations

PRP technology is implemented in IACS devices, therefore network infrastructure devices (other than the RedBoxes) do not have to be PRP capable. PRP is not dependent on any particular LAN topology and should provide a single fault tolerance with zero data loss even with non-resilient topologies in each LAN such as star, linear, or a single switch.

When designing a PRP network, follow these recommendations:

  • If possible, use resilient topologies (redundant star, ring) in each LAN for additional resiliency protection. In this case, an extended outage or maintenance in one of the LANs still should allow the IACS to recover from a subsequent fault in the other LAN (with convergence time depending on the resilient LAN protocol).
  • Design the architecture to avoid or minimize architecture-wide faults that impact both LANs such as power failures or damage of both redundant cabling paths. Use redundant power sources and physically isolated cabling conduits for each LAN.
  • Help protect the network from uncontrolled Ethernet loops that may cause broadcast or multicast storms. Follow best practices and use recommended topologies for resiliency protocols that may be used in LAN A and LAN B (e.g., Spanning Tree, DLR, REP). Do not disable loop prevention mechanisms on infrastructure IES.
  • Use the same or similar topologies for both LANs with comparable network latency and number of hops in normal network conditions. Avoid using different types of connectivity between LAN A and LAN B, for example a high-speed wired network for LAN A and low bandwidth, high-latency wireless technology for LAN B.
  • Do not connect IES (other than RedBoxes) to both LAN A and LAN B. Direct links between LAN A and LAN B IES are not allowed.
  • Do not connect any RedBox IES to each other via a Layer 2 path that bridges any of the VLANs that exist in the PRP network. Such a connection creates a bridging loop in the network. Layer 3 routed paths are allowed.
  • If routing is required in the PRP network, configure a RedBox IES as the router. Do not enable routing on the LAN A or LAN B IES. For recommendations on the routing redundancy design with PRP and how to connect to the Industrial Zone network, see Connectivity to the Industrial Zone Network.
  • Apply the same recommended network and security practices as for a non-PRP network, such as using managed switches with diagnostic, loop prevention, multicast management and security features, minimizing broadcast domains with VLAN segmentation, hardening the network and the IACS applications against security threats, maintaining good change control practices, and IES configuration management.

Parallel Redundancy Protocol Topology Examples

This section provides some examples of PRP architectures and topologies.

Figure 2-5 shows an example of PRP deployment with two parallel fault-isolated physical paths. This could be useful in mining or transportation applications (e.g., parallel tunnels), marine applications (two sides of a ship), and other similar use cases.

In the example below, both LANs use a ring topology rather than linear topology for additional resiliency. In most greenfield installations, the cost of having a return cable path is insignificant when installing a cable bundle. The benefits of using a resilient LAN topology are greater than the additional effort of configuring and monitoring of a ring protocol.

The example architecture also implements redundant programmable automation controllers (PAC) with ControlLogix ® redundancy for greater availability and protection from controller faults.

Figure 2-5 PRP Topology Example with Parallel Paths

 

387868.jpg

Figure 2-6 shows an example of a PRP topology with dual rings and a single (non-redundant) PAC. A dual-ring topology is common for water/wastewater, mining, oil and gas, and other industries that traditionally have used redundant dual-media topologies over large geographical areas.

Figure 2-6 PRP Topology Example with Dual Rings—Single PAC

 

387869.jpg

Figure 2-7 shows an example of a PRP topology with dual rings and ControlLogix redundant PACs.

Figure 2-7 PRP Topology Example with Dual Rings—Redundant Controllers

 

387870.jpg

Figure 2-8 shows an example of the star topology in a PRP network.

Note that access IES could also be connected with redundant uplinks to the aggregation IES in the LAN A or B, for example using EtherChannel technology. The cost of additional cabling and available ports should be considered.

Figure 2-8 PRP Star Topology Example

 

387871.jpg
note.gif

Noteblank.gif The CPwE PRP architecture has been tested and validated using a star, redundant star, and dual ring topology for LAN A and LAN B with a mix of redundant and non-redundant PACs.


Connecting HMI and PCs to a PRP Network

PCs, HMI terminals and thin clients with a single Ethernet port can be connected to a PRP network:

  • As a SAN to a LAN A or LAN B (two terminals with identical content can be connected to each LAN for redundancy)
  • As a VDAN to a RedBox

PCs with NIC Teaming or thin clients with redundant Ethernet ports in the Active/Standby mode can be connected as a VDAN to one or two RedBoxes using two Ethernet links. See Figure 2-9 below.

  • Do NOT connect devices with redundant NICs without native PRP support to LAN A and LAN B switches. Doing so may cause significant delays for HMI traffic during the switchover between the NICs.
note.gif

Noteblank.gif Validating third-party network adapters with native PRP support is out of scope for CPwE PRP.


Figure 2-9 Connecting PC and HMI Terminals

387872.jpg

Using Wireless Media with PRP

PRP over wireless can, in principle, be supported when:

  • Both LANs use the same wireless technology with similar latency
  • Separate wireless channels are used for LAN A and LAN B links
  • Radios and other infrastructure equipment either support PRP as a feature or can forward Ethernet frames with PRP trailer without modification

It is highly recommended to validate PRP operation with the selected wireless equipment and verify compatibility with the vendor before deployment.

note.gif

Noteblank.gif PRP over wireless is out of scope for this release of CPwE PRP architecture.


Unsupported Topologies

This section describes a number of invalid PRP topologies or topologies that are not recommended due to performance or availability concerns.

  • LAN A and LAN B infrastructure cannot be bridged together using a direct link, a non-RedBox IES, or a 2-port embedded switch device that does not support PRP (e.g., a ControlLogix 1756-EN2TR module or a 1783-ETAP EtherNet/IP DLR tap module). See Figure 2-10 below.

Figure 2-10 Invalid Topology—Bridging LAN A and LAN B

 

387873.jpg
  • RedBox IES cannot be connected through non-PRP ports via a Layer 2 path that forwards traffic from any of the PRP VLANs, including IACS data VLANs, management VLAN, or the native VLAN. See Figure 2-11 below.

Figure 2-11 Invalid Topology—Bridging PRP VLAN via RedBox non-PRP Ports

 

387874.jpg
  • LAN A and LAN B topologies should not contain 2-port embedded switch devices, including 1783-ETAP modules, in the data path. Embedded switch devices cannot be configured for the larger MTU sizes to accommodate the PRP trailer in the frame. As a result, maximum size Ethernet frames may be dropped. This also applies to any IES without the option to increase the MTU size (Figure 2-12).

Figure 2-12 Unsupported Topology—Traversing 2-port Embedded Switch Devices

 

387875.jpg
  • It is not recommended to combine high-bandwidth low-latency LAN as the primary LAN and low-bandwidth high-latency WAN or wireless technology as the secondary LAN. One of the possible issues could be increased chance of duplicate frames arriving late and being wrongly accepted as non-duplicate (Figure 2-13).

Figure 2-13 Unsupported Topology—Using High-latency Connection as Secondary

 

387876.jpg

Connectivity to the Industrial Zone Network

The CPwE PRP architecture provides guidelines for connecting a PRP-enabled Cell/Area Zone to the plant-wide or site-wide network in the Industrial Zone.

Although IACS applications may exist when a PRP network is deployed as a standalone network (e.g., an isolated I/O network), having plant-wide or site-wide connectivity to the IACS network with PRP technology allows many benefits of the converged network model:

  • Remote access for diagnostics and troubleshooting
  • Distributed network applications using virtual server environment in the Level 3 Site Operations
  • Access to IACS device data and analytics from the Enterprise Zone and / or the cloud as part of the Connected Enterprise® system™.

When connecting a PRP topology with two redundant and isolated LANs to a non-PRP resilient topology, these rules should be followed to avoid bridging loops:

  • Only RedBox IES can be used as gateways from a PRP to a non-PRP part of the network.
  • Any non-PRP enabled path between RedBox IES should only include Layer 3 (routed) connections.

Figure 2-14 shows the CPwE PRP architecture that provides redundant connectivity from a resilient distribution layer in the Industrial Zone to a pair of distribution Layer 3 RedBox IES connected to the PRP Cell/ Area Zone with a star LAN topology.

Figure 2-14 Connectivity to the Industrial Zone

 

387877.jpg
  • Redundant Layer 3 RedBox IES are configured with Hot Standby Router Protocol (HSRP) as active/standby default gateways for IP subnets in the PRP topology.

blank.gif Layer 3 RedBox IES use PRP channel ports to send HSRP hello packets and monitor redundancy state.

  • Redundant links between Layer 3 RedBox IES and uplinks to the distribution stack are configured as Layer 3 EtherChannels (routed ports).
  • Dynamic routing protocol is configured between the distribution Layer 3 RedBox IES with HSRP pair and the distribution switch stack.

blank.gif Enhanced Interior Gateway Routing Protocol (EIGRP) has been validated as part of the CPwE PRP.

blank.gif Open Shortest Path First (OSPF) routing protocol can also be used depending on the existing infrastructure and requirements. CPwE PRP has not been validated with OSPF.

blank.gif Static routes between RedBox IES and the distribution / core layer are allowed but not recommended due to anincreased complexity of configuration and maintenance in large environments. CPwE PRP has not been validated with static routing.

  • Cisco StackWise-480 technology has been tested with the distribution switches for redundancy.

blank.gif Other resiliency protocols and technologies can be used as outlined in the CPwE Resiliency DIG but have not been tested and validated as part of this CPwE PRP.

  • Small-scale deployments can combine core and distribution layers into one redundant switch layer.

Figure 2-15 shows a similar CPwE PRP architecture with redundant connectivity via a pair of Layer 3 RedBox IES where DLR is used in the PRP Cell/Area Zone as a ring LAN topology.

Figure 2-15 Connectivity to the Industrial Zone (DLR LAN Topology)

387878.jpg

EtherChannel, HSRP, and Routing Protocol Considerations

General EtherChannel and HSRP recommendations and configuration guidelines are provided in the CPwE Resiliency Design and Implementation Guide:

For information about EIGRP design and configuration, refer to:

For information about OSPF, refer to:

note.gif

Noteblank.gif HSRP and EIGRP features require Layer 3 firmware level for Stratix IES and Network Advantage license type for Cisco Catalyst 9300 switches.


An important consideration for the CPwE PRP routed design is that both Layer 3 RedBox IES carry traffic from the Industrial Zone core network to the Cell/Area Zone, e.g., from an HMI server to an HMI client or a controller (Figure 2-16).

This is due to the equal cost routing in EIGRP or OSPF on the distribution switch where downstream traffic is split roughly evenly between the links to the RedBoxes. As a result, a failure of either the active or the standby HSRP RedBox IES may impact the IACS traffic from Level 3 Site Operation. Similarly, restoring a RedBox IES as the standby HSRP gateway may lead to comparatively small convergence times as the traffic is restored on the second path.

Figure 2-16 Routed Traffic with PRP

 

387879.jpg

The recommended and validated EtherChannel, HSRP, and EIGRP configuration for CPwE PRP is provided in Chapter3, “CPwE Parallel Redundancy Protocol Configuration”

Routed Traffic Convergence

PRP provides zero-loss redundancy and single-fault tolerance for Layer 2 (Ethernet) traffic in the PRP network, including protection against a LAN switch fault or a link fault affecting a DAN, a LAN switch or a PRP channel port on a RedBox.

Layer 3 (routed) traffic traverses a Layer 3 switch (the default gateway) which must be a RedBox IES. CPwE PRP architecture includes two redundant Layer 3 RedBox IES to provide resiliency for the routed traffic.

note.gif

Noteblank.gif PRP does not provide zero-loss redundancy for routed traffic when a Layer 3 RedBox fails. Routed traffic is interrupted and reconverges during faults impacting the Layer 3 RedBox IES or routed uplinks connected to the RedBoxes.


Depending on the fault type, convergence time for routed IACS data may include:

  • HSRP failover time between Layer 3 RedBox IES
  • Dynamic routing protocol convergence
  • EtherChannel failover
  • Layer 2 switched network convergence (e.g., MAC table updates)

Different routing configurations, protocols and distribution switch platforms may provide different results. Table 2-2 summarizes various faults and the observed impact on the routed traffic in the CPwE PRP testing.

 

Table 2-2 Routed Traffic Convergence 1

Event Type

Layer 3 traffic to/from PRP Cell/Area Zone

Layer 3 traffic between VLANs in PRP Cell/Area Zone

Convergence Time

Typical Time

Convergence Time

Typical Time

Layer 3 EtherChannel link loss or restore

EtherChannel

< 150ms

N/A

N/A

Layer 3 EtherChannel loss (both links)

Routing protocol

1 - 3 seconds

N/A

N/A

Layer 3 RedBox down (HSRP Active)

HSRP and routing protocol

1 - 3 seconds

HSRP

1 - 2 seconds

Layer 3 RedBox down (HSRP Standby)

Routing protocol

1 - 3 seconds

N/A

N/A

Layer 3 RedBox restored (becomes HSRP Standby)

Routing protocol

< 1 second

N/A

N/A

LAN switch fault

PRP

0

PRP

0

Link fault in the PRP LAN

PRP

0

PRP

0

Link fault between Layer 3 RedBox and LAN switch

PRP

0

PRP

0

1.With HSRP hold timers 750 ms, EIGRP routing protocol and LACP Active EtherChannel mode.

note.gif

Noteblank.gif It is important to evaluate IACS application requirements for the routed traffic and compare with the expected convergence times. For example, timeout period with typical Requested Packet Interval (RPI) values for routed CIP Class 1 data (Produced/Consumed tags) may be lower than the HSRP convergence. As a result, Produced/Consumed connections may be dropped during a Layer 3 RedBox failure.


Connectivity to Device Level Ring

The ODVA, Inc. Device Level Ring (DLR) resilient LAN technology is designed to provide ring topology resiliency for critical IACS applications. DLR supports fast ring convergence (single-fault tolerant) in the event of an IACS device or link failure.

For more information on DLR in CPwE, refer to:

This section describes recommendations for connecting an existing DLR topology without PRP to a PRP architecture.

  • The recommended resilient architecture is to connect the DLR topology via a separate distribution switch (Figure 2-17) below.

blank.gif A controller chassis can be connected to both PRP and DLR networks using separate Ether-Net/IP modules in the chassis. In this example, a ControlLogix Redundancy chassis is con-nected to the PRP topology and the switch DLR topology.

Figure 2-17 Connecting Controller Chassis to both PRP and DLR

 

387880.jpg
  • An alternative non-redundant option is to use a RedBox IES as the DLR supervisor for one or multiple rings (Figure 2-18) below. DLR ports cannot be the same as the PRP channel ports.
note.gif

Noteblank.gif In this case the RedBox is a single point of failure for the traffic between the DLR and the PRP topologies. This architecture is not recommended for critical IACS data traversing the RedBox and has not been validated for performance or scalability in this CPwE PRP.


Figure 2-18 PRP to DLR Connectivity via a Single RedBox

 

387881.jpg
  • Using two RedBox IES as redundant DLR gateways is not supported due to increased convergence time for traffic traversing the gateways during certain faults, which exceeded requirements for IACS applications (Figure 2-19) below.

Figure 2-19 Unsupported Topology—PRP to DLR Redundant Gateways

 

387882.jpg
note.gif

Noteblank.gif An EtherNet/IP module in the PRP mode cannot be used as part of the DLR or a linear topology. PRP-enabled modules do not implement the embedded switch technology and traffic cannot traverse from port A to port B. Similarly, an Ethernet module in the DLR mode cannot be connected to both LAN A and LAN B.


 

Network Services Recommendations

This section provides recommendations for network services and IES features that could be required in the PRP-enabled IACS network, such as VLAN segmentation and trunking, multicast traffic management, time synchronization, and Network Address Translation (NAT).

VLAN Segmentation and Trunking

PRP technology can be deployed in networks with VLAN segmentation. Links between IES can be configured with VLAN trunking to carry traffic from DANs and VDANs that belong to multiple VLANs (Figure 2-20) below.

Figure 2-20 VLAN Segmentation (Zoning) in PRP Network

 

387883.jpg

Follow these recommendations when using VLAN segmentation with PRP:

  • Both PRP ports on a DAN should be connected to the same VLAN in LAN A and LAN B.
  • The PRP channel ports on a RedBox IES can be configured as VLAN trunk ports (more common) or access mode ports (i.e., single VLAN). Trunk mode allows having VDANs in multiple VLANs or using a separate management VLAN for the RedBox IES.
  • PRP channels on the redundant Layer 3 RedBox IES in CPwE PRP should be configured as VLAN trunks. The architecture has been validated with inter-VLAN IACS traffic which is routed through the Layer 3 RedBox with HSRP.

blank.gif Traffic between VLANs (Layer 3) is impacted if the active HSRP gateway fails. Convergence time depends on the HSRP parameters.

  • Links between IES in each PRP LAN should be configured as VLAN trunks. Per CPwE best practice, the native VLAN should be different from any of the IACS VLANs
note.gif

Noteblank.gif PRP supervisory frames are Layer 2 multicast Ethernet frames that cannot be routed between VLANs. As a result, DANs can only report diagnostic information about PRP devices in their VLANs.


For network architectures with two PRP channels configured on the same pair of Layer 3 RedBox IES, configure VLANs as follows to avoid network loops (Figure 2-21) below.

  • Use separate sets of data, management and native VLANs in Cell/Area Zones connected to each PRP channels
  • Configure the allowed VLAN list on the trunk ports in the PRP channel group (VLAN pruning), making sure that VLANs from the other PRP channel are not allowed.

Figure 2-21 Two PRP Channels with VLAN Pruning

387884.jpg

Spanning Tree Protocol

Design and configuration of the Spanning Tree Protocol (STP) in PRP LAN A and LAN B should follow general recommendations in the CPwE Resiliency Design and Implementation Guide. Special considerations exist for STP operation between a RedBox IES and infrastructure IES (Figure 2-22) below.

  • A RedBox IES serves as a boundary between separate STP instances in LAN A and LAN B. There should be no common STP domain bridging two redundant LAN A and LAN B.
  • Bridge Protocol Data Unit (BPDU) frames from STP are filtered on the PRP channel ports. As a result, LAN A and LAN B switches exclude the RedBox IES in the STP operation.
  • STP is running on the RedBox IES by default and should be kept enabled for loop prevention on the non-PRP ports.

blank.gif Two RedBox IES cannot be connected to each other via non-PRP ports. Doing so will cause a bridging loop.

  • PortFast Trunk mode should be configured for the PRP channel group on all RedBox IES, including redundant Layer 3 RedBox IES, and on the LAN A and LAN B IES ports connected to the RedBox. This is necessary to minimize port recovery time during network faults.
  • Best practices for STP should be followed within each LAN, such as explicitly configuring the primary and secondary STP root bridge, using recommended redundant star topologies, and avoiding complex meshed topologies.

Figure 2-22 Spanning Tree Operation in the PRP Network

 

387885.jpg
note.gif

Noteblank.gif Other resiliency protocols such as DLR and REP, if present in the PRP LAN topologies, may have their own considerations for STP interoperability, separate from the PRP considerations. Refer to the corresponding CPwE DLR Design Guide and CPwE Resiliency Design and Implementation Guide for more information.


Multicast Management

Multicast EtherNet/IP traffic is required for I/O and consumed data in ControlLogix Redundancy and for CIP Sync communication using Precision Time Protocol (PTP). Both types of multicast data could be used in IACS applications where PRP technology is deployed.

It is critical to follow design and configuration guidelines for multicast traffic management in CPwE PRP to make sure that high availability is achieved:

  • Enable Internet Group Management Protocol (IGMP) snooping on all IES to reduce the amount of unnecessary multicast traffic to end nodes. IGMP snooping is enabled by default after Express Setup on Stratix managed switches.
  • Configure IGMP querier on the redundant Layer 3 RedBox IES with HSRP. Make sure that at least two IGMP queriers are present.

blank.gif In order to win the querier election, switches should have the lowest IP addresses in the subnet.

blank.gif For networks without routing (not common), a Layer 2 RedBox can be used as a querier

  • Disable IGMP querier on each IES in LAN A and LAN B.
  • Configure uplink ports on the LAN IES as static multicast router (mrouter) ports, specifically the ports that could be in the path to the IGMP querier (distribution RedBox IES). This configuration enables multicast traffic flow in the topology when the IGMP querier changes (for example when the active HSRP gateway reboots).

Figure 2-23 below illustrates IGMP snooping configuration in the PRP topology. For details on how to configure these settings, refer to Chapter3, “CPwE Parallel Redundancy Protocol Configuration”

Figure 2-23 Multicast Management with PRP

 

387886.jpg
note.gif

Noteblank.gif After a LAN in a PRP network encounters a fault and is then repaired, there could be a delay until the IGMP querier reinstates the multicast traffic in the recovered LAN (typically within 2 minutes after the LAN is repaired). During that time, the other LAN will continue forwarding multicast traffic, however, PRP redundancy is not provided for multicast data for a short period of time.


Network Address Translation

Network Address Translation (NAT) feature in IES provides benefits to OEM machine or process skid builders, such as IP address reuse and commissioning of “cookie cutter” machines without reprogramming, easier maintenance of machine configurations and controller programs, and better traffic control and additional security by limiting access only to selected devices on a machine or skid. It may also help plant or site engineers with integrating legacy stand-alone equipment into the plant-wide or site-wide network.

PRP technology is compatible with NAT since PRP operates in Layer 2 and is transparent to the higher layers of the network stack where IP addresses are used.

There are two possible implementations of NAT in a PRP topology (Figure 2-24) below:

  • Configure NAT on the LAN A and LAN B infrastructure IES to provide translation for DANs.

blank.gif NAT configurations must match exactly between switches

  • Configure NAT on the RedBox IES to provide translation for VDANs.

Figure 2-24 Network Address Translation with PRP

 

387887.jpg

Since CPwE PRP implements HSRP for router (default gateway) redundancy, the following translation rules need to be added to the NAT configuration:

  • Gateway translation for the virtual IP address of the HSRP gateways
  • Public-to-Private translation for the physical IP addresses of both active and standby HSRP gateways (RedBox IES)

Other general NAT recommendations and limitations may apply, for example topology considerations, multicast restrictions, or application restrictions. For more information on NAT with Stratix IES, refer to:

Dynamic Host Configuration Protocol

Dynamic Host Configuration Protocol (DHCP) assigns IP address information from a pool of available addresses to newly connected devices (DHCP clients) in the network. In industrial networks that use DHCP, managed IES are typically configured with DHCP persistence to assign IP addresses to IACS devices based on a specific port.

DHCP is supported in the PRP topology, including DHCP Persistence.

  • DHCP Persistence can be configured on the Layer 2 RedBox IES to automatically assign IP addresses to VDANs based on the port.
  • DHCP Persistence can be enabled on the infrastructure IES to assign IP addresses to DANs based on the port, as long as:

blank.gif DHCP configuration on LAN A and LAN B switches matches exactly including the DHCP pool range and the reserved addresses per port

blank.gif A DAN is connected to the same port number on each LAN IES.

blank.gif Reserved Only setting is enabled for the DHCP scope.

  • DHCP configuration on the infrastructure IES without DHCP Persistence is not supported. The risk exists of assigning duplicate IP addresses for DANs.
  • Using DHCP for SANs is not recommended. If such configuration is required, DHCP scopes in each LAN should not overlap.
  • DHCP Snooping should be enabled for the DHCP scope configured on the infrastructure and RedBox IES

Time Distribution in CPwE PRP

This section provides recommendations for plant-wide or site-wide time distribution in the CPwE PRP architecture.

IACS devices use ODVA, Inc. CIP Sync technology based on the IEEE 1588™ Precision Time Protocol (PTP) to synchronize clocks in the control system. CIP Sync is designed for local and plant-wide or site-wide IACS applications requiring very high accuracies.

Industrial computers, application servers, managed IES and other network devices use Network Time Protocol (NTP) for timestamping Factory Talk® Alarms and Events.

For more information on CIP Sync, refer to: https://www.odva.org/technology-standards/distinct-cip-services/cip-sync

For general information on designing and configuring time distribution using CIP Sync (PTP) and NTP in the plant-wide or site-wide network, refer to:

Precision Time Protocol (CIP Sync) with PRP

PRP networks support CIP Sync by implementing the doubly-attached clock model as specified in IEC 62439-3 standard (Figure 2-25):

  • In a DAN or a RedBox with the master clock role, both ports A and B operate as CIP Sync master ports in their LAN segments.
  • In a DAN or a RedBox with a slave clock role, both ports A and B are paired and function as CIP Sync slave ports.

blank.gif One port is the active slave port that synchronizes to the master clock, measures path delay, and tunes the clock. This port reports its state as SLAVE.

blank.gif The other port is passive and reports its state as PASSIVE_SLAVE or PASSIVE. The passive slave port also measures path delay and maintains close synchronization to the master clock. In case of a network failure on the active slave port, the passive slave port clock transitions smoothly to the active state.

blank.gif Either LAN A or LAN B port can be chosen as active slave port depending on the Best Master Clock Algorithm (BMCA) of the PTP protocol. As a result, some DANs can be syncing over LAN A and some over LAN B infrastructure.

  • PTP traffic in the PRP network is handled differently from non-PTP traffic:

blank.gif Ethernet frames that carry PTP information are not duplicated and exclude the redundancy trailer.

blank.gif PTP packets from DANs, such as Sync, Delay Request and Delay Response, are generated independently for LAN A and LAN B ports.

Figure 2-25 PRP Doubly Attached Clock Model

 

387888.jpg

In the PTP architecture, all clocks are synchronized to a Grandmaster clock (GM). The GM must be a DAN or a VDAN and cannot be a SAN attached to LAN A or LAN B. Figure 2-26 shows some of the options for a GM in the PRP topology:

  • A PAC in a chassis with an EtherNet/IP PRP module (DAN)
  • A PAC (VDAN) connected to a RedBox IES
  • A Global Positioning System (GPS) time module (VDAN) connected to a RedBox IES
  • A RedBox IES configured as a GM in the NTP/PTP mode

Figure 2-26 Grandmaster Clock in a PRP Topology

387889.jpg

For critical applications using time synchronization:

  • Use primary and secondary grandmasters for resiliency
  • Do not connect grandmasters as SANs (one in LAN A, another in LAN B).
  • Implement measures to reduce the risk of grandmasters failing such as providing backup power to the grandmasters and network infrastructure
  • Reduce the risk of simultaneous failures for the primary and secondary grandmasters by using separate network switches, redundant power supplies, and using multiple reference time sources.
note.gif

Noteblank.gif Switchover from a primary GM to a backup GM is a system-wide event and may cause disruptions to time synchronization.


PTP Recommendations for IES

Use these guidelines for configuring IES in the CPwE PRP architecture with PTP:

  • Configure Layer 2 RedBox IES as boundary clocks (BC).
  • Configure Layer 3 RedBox IES as boundary clocks or as NTP/PTP clock (if RedBox IES are used as the active and backup GM).
note.gif

Noteblank.gif Transparent clock (TC) mode or forward mode are not supported on a RedBox IES.


  • Configure infrastructure IES in LAN A and LAN B as boundary clocks if time synchronization is required in multiple VLANs across PRP Cell/Area Zone

blank.gif Use VLAN trunking for Layer 2 links between IES and make sure that the Native VLAN matches on each end of the trunk link.

  • Transparent clock mode can be used on infrastructure IES in LAN A and LAN B when only one VLAN is used for time synchronization between IACS devices connected to TC switches.

blank.gif Transparent clock IES do not propagate PTP information between VLANs.

blank.gif PTP VLAN ID should be configured on the boundary clock IES for ports connected to the transparent clock IES.

note.gif

Noteblank.gif Infrastructure IES without PTP support, or in the PTP forward mode, are allowed but not recommended due to lower time synchronization accuracy in the PRP architecture.


CPwE PRP architecture with GPS Reference Clock

Figure 2-27 shows an example of the recommended and validated CPwE PRP architecture with redundant GPS time modules connected to the distribution switch stack.

  • Time modules synchronize to GPS satellites as reference clocks and distribute time information via PTP and NTP to the plant-wide or site-wide network.
  • One of the time modules is the primary GM and the primary NTP server. The second module serves as a secondary GM / NTP server in case the first module becomes unreachable.

blank.gif Time modules should be connected to different switches in the distribution stack and should use separate power supplies and preferably separate GPS antennas.

  • Cisco Catalyst 9300 switches in the distribution stack and Layer 3 RedBox IES are configured in the BC mode.
note.gif

Noteblank.gif PTP BC mode is supported with EtherChannels on the Cisco Catalyst 9300 starting with IOS XE 17.2.1 and on Stratix IES starting with IOS 15.2(7)E3


  • Infrastructure IES in LAN A and LAN B are configured in the BC mode to enable time synchronization in multiple VLAN.

Figure 2-27 Example of CPwE PRP architecture with PTP CIP Sync - Redundant GPS Modules

 

387890.jpg
note.gif

Noteblank.gif Managed IES and other network devices use NTP for timestamping network events in logs. Both NTP and PTP must be configured in IES to correlate all events logged by the network infrastructure devices with IACS events.


Below are considerations and recommendations based on the CPwE PRP testing:

  • In steady state with no network faults, time synchronization between IACS devices was maintained within the resolution of the CIP Sync modules used in the testing (< 8 microseconds).
  • Switchover from the primary to secondary GM may cause temporary degradation of time accuracy for IACS devices. Time skew as much as 2 milliseconds between IACS devices was observed for a period of several seconds.
  • Network events that change master clock for BC switches (LAN or RedBox) or change the active slave port on a DAN can cause similar PTP disturbances. This may include HSRP failover between Layer 3 RedBox IES, LAN A or LAN B switch faults, and various link faults in the network.
  • It is recommended to use latest hardware series and firmware revisions for IACS devices and IES.
  • 1756-EN4TR ControlLogix Ethernet modules (firmware 4.001 and later) and FLEX 5000 EtherNet/IP adapters are recommended for better CIP Sync performance during network faults.

Examples of PRP Architectures using NTP/PTP mode

If GPS time source is not available, time modules or Layer 3 RedBox IES can be configured as Grandmaster clocks in the NTP/PTP mode. In this case, time modules or RedBoxes use NTP servers in the Industrial or Enterprise Zone as reference clocks and provide PTP synchronization for the IACS devices in one or multiple Cell/Area Zones.

Figure 2-28 below shows an example of a PRP architecture with redundant time modules in NTP/PTP mode (without GPS) connected to the distribution switch stack. In this example, the time modules are the primary and secondary PTP GM and are the NTP servers for infrastructure devices in the Cell/Area Zone.

Figure 2-28 Example of PRP Architecture with Redundant NTP/PTP Module

387891.jpg

Figure 2-29 shows an example of a PRP architecture with Layer 3 RedBox IES in NTP/PTP mode. In this case, NTP is used for time distribution in the plant-wide or site-wide network, and PTP is used in each Cell/Area Zone with NTP/PTP configuration on IES.

Multiple Cell/Area Zones with PRP use separate pairs of Layer 3 RedBox IEs for NTP to PTP transition. As a result, time synchronization accuracy between IACS devices in different Cell/Area Zones depends on the accuracy that NTP can provide.

Figure 2-29 Example of PRP Architecture with NTP/PTP on RedBox IES

387892.jpg

In both NTP/PTP architectures above, using NTP servers as reference clocks instead of GPS limits the accuracy of time synchronization. The system behavior can be impacted by network performance and availability of the NTP time sources. For example, a fault of the active NTP server and switchover to another NTP server may impact PTP time synchronization for IACS devices.

Applying PRP with IACS Applications

PRP technology is independent from the network layer or IACS application layer and therefore can be used with different types of IACS applications that require high availability. The following IACS applications and data types have been tested and validated within the CPwE PRP:

  • CIP Class 1 I/O and Produced Consumed tags (unicast and multicast)
  • ControlLogix Redundancy (requires version 31.052 or higher)
  • CIP Class 3 messaging
  • CIP Safety™
  • CIP Sync and IEEE 1588 Precision Time Protocol (PTP)
  • FactoryTalk View Site Edition
  • FactoryTalk View Machine Edition
  • FactoryTalk Linx
  • Studio 5000 Logix Designer®

ControlLogix Redundancy with PRP

A ControlLogix redundancy system provides greater availability by establishing redundancy between a pair of controller chassis with identical components. ControlLogix redundancy is further enhanced by using high availability networks, such as PRP, to provide fault tolerance in the infrastructure.

CPwE PRP architecture has been tested with 5570 and 5580 ControlLogix Redundancy including redundant PAC chassis with EtherNet/IP modules (DAN) communicating to DAN or VDAN I/O devices, other PACs, and FactoryTalk applications. ControlLogix Redundancy system configuration included the following:

  • EtherNet/IP PRP modules for I/O and Produced Consumed data configured for IP address swapping during a chassis switchover
  • Dedicated EtherNet/IP PRP modules for HMI data that do not swap IP addresses
  • Redundant ControlLogix Controller shortcut type in FactoryTalk Linx that provides paths to the Primary and Secondary controllers through the PRP modules with no IP address swapping.
  • ControlLogix Redundancy firmware revision 31.052 or later; FactoryTalk Linx 6.11 or later.
  • Infrastructure and RedBox IES configured for IGMP snooping as described earlier
note.gif

Noteblank.gif Routed traffic to and from the ControlLogix Redundancy (e.g., FactoryTalk data or CIP Class 3 messages between VLANs) can be affected by Layer 3 switch faults, HSRP and routing protocol convergence. These types of faults are not covered by the PRP zero data loss mechanism and should be considered independently in the PRP architecture design calculations.


Figure 2-30 shows an example of a 5580 ControlLogix Redundancy system using PRP:

  • 1756-EN4TR modules with PRP (firmware 4.001 or later) in the ControlLogix redundant controller chassis (version 34 or later)
  • One pair of modules is dedicated for FactoryTalk Linx communication (no IP swapping)
  • The second pair of modules is configured with IP swapping to support multicast I/O and Produce Consume traffic
  • Single or redundant 1756-EN4TR modules with PRP in the remote I/O chassis

Figure 2-30 Example of 5580 ControlLogix Redundancy with PRP

387893.jpg

 

For more information on using PRP with ControlLogix Redundancy, refer to:

PlantPAx Distributed Control System with PRP

The PlantPAx® Distributed Control System is an integrated control and information solution that provides Plantwide Optimization for a wide range of industries. This single-platform system is built on open industry standards to help support the seamless integration of system components, and to provide connectivity to high-level business systems.

PRP is the recommended redundancy technology for a PlantPAx system where the highest level of availability is required. A PlantPAx system design uses a CPwE PRP architecture to build a redundant network topology with infrastructure duplication, fault tolerance capability, zero recovery time within the PRP zone, and minimal recovery time for traffic leaving the PRP zone.

For more information on PlantPAx system infrastructures with PRP, refer to: