Scalable Time Distribution within a Converged Plantwide Ethernet Architecture
The prevailing trend in Industrial Automation and Control System (IACS) networking is the convergence of technology, specifically IACS operational technology (OT) with information technology (IT). Converged Plantwide Ethernet (CPwE) helps to enable IACS network and security technology and OT-IT persona convergence through the use of standard Ethernet, Internet Protocol (IP), network services, security services, time synchronization technologies, and EtherNet/IP. A real-time converged plant-wide IACS architecture helps to enable the Industrial Internet of Things (IIoT).
Business practices, corporate standards, policies, industry standards, and tolerance to risk are key factors in determining the degree of time synchronization required within a plant-wide IACS architecture. IACS networks differ from their IT counterparts in their need to support significantly lower latency (time delay between message sent and message received) and jitter (the variance of the latency) to help enable real-time IACS communications. Time synchronization helps to support time-critical data for the most demanding IACS applications.
Successful deployment of IIoT IACS applications within CPwE Architectures (Figure 1) depends on a network infrastructure design that addresses IACS application performance requirements. The content of CPwE, which is relevant to both OT and IT disciplines, consists of documented architectures and key tenets from OT and IT to help achieve real-time communications to support IIoT IACS applications. CPwE key tenets include:
- Smart IIoT Devices—controllers, I/O, drives, instrumentation, actuators and analytics
- Zoning (segmentation)—smaller connected LANs, functional areas and security groups
- Managed Infrastructure—managed industrial Ethernet switches (IES) and industrial firewalls
- Resiliency—robust physical layer and resilient or redundant topologies with resiliency protocols
- Time-critical Data—data prioritization and time synchronization via CIP Sync™ and IEEE-1588 Precision Time Protocol (PTP)
- Wireless—unified wireless LAN (WLAN) to enable mobility for personnel and equipment
- Holistic Defense-in-Depth Security—multiple layers of diverse technologies for threat detection and prevention, implemented by different persona (e.g., OT and IT) and applied at different levels of the plant-wide IACS architecture
- Convergence-ready—seamless plant-wide integration by trusted partner applications
The ODVA, Inc. CIP Sync technology uses the Common Industrial Protocol (CIP™) application layer protocol and the IEEE 1588-2008 Precision Time Protocol (PTP) standard for time synchronization. CIP Sync and IEEE 1588-2008 are designed for local and plant-wide IACS applications requiring very high accuracies beyond those attainable with Network Time Protocol (NTP).
Deploying Scalable Time Distribution within a Converged Plantwide Ethernet Architecture Design Guide (CPwE Time) outlines several use cases for designing and deploying IEEE 1588 PTP and CIP Sync technology throughout a plant-wide IACS network infrastructure. CPwE Time was architected, tested and verified by Cisco Systems and Rockwell Automation with assistance by Panduit.
CPwE Time Solution Use Cases
CPwE is the underlying architecture that provides standard network and security services for control and information disciplines, devices and equipment found in modern IACS applications. The CPwE architectures (Figure 1) were architected, tested and validated to provide design and implementation guidance, test results and documented configuration settings. This can help to achieve the real-time communication, reliability, scalability, security and resiliency requirements of modern IACS applications.
An IACS is deployed in a wide variety of discrete and process manufacturing industries such as automotive, pharmaceuticals, consumer packaged goods, pulp and paper, oil and gas, water/wastewater, mining and energy. IACS applications are made up of multiple control and information disciplines such as continuous process, batch and discrete and hybrid combinations. One of the challenges facing industrial operations is the industrial hardening of standard Ethernet and IP-converged IACS networking technologies to take advantage of the business benefits associated with IIoT.
The Deploying Scalable Time Distribution within a Converged Plantwide Ethernet Architecture Design Guide outlines the concepts, requirements and technology solutions for reference designs developed around a specific set of priority use cases. These use cases were architected and tested for solution functional verification with limited scale by Cisco Systems and Rockwell Automation with assistance by Panduit to help support time synchronization within a converged plant-wide EtherNet/IP IACS architecture.
The CPwE Time Design Guide includes:
–
IEEE 1588 Precision Time Protocol (PTP)
– Distributed Motion (not included in CPwE Time)
– Design, configuration and diagnostic considerations for plant-wide (Levels 0-3) IEEE 1588 PTP and CIP Sync deployments
– Limited resiliency and reliability PoC testing
– Layer 2 IES—Allen-Bradley® Stratix 5700/5400
– Layer 3 IES—Allen-Bradley Stratix 5410
Note This release of the CPwE architecture focuses on EtherNet/IP connectivity, which uses the ODVA, Inc. Common Industrial Protocol (CIP) and is ready for the Industrial Internet of Things (IIoT). For more information on EtherNet/IP and CIP Sync, see odva.org at the following URL:
CPwE Time Architecture Overview
ODVA, Inc. CIP Sync is used across a broad range of IACS applications to synchronize control system clocks (Figure 2) and helps enable applications such as event sequencing and logging. For example:
- A sequence of events or first fault system can use timestamps to determine the order in which faults occurred in the system. This allows for the tracking of faults to establish the first in a chain of faults. These types of applications use dedicated alarm instructions in the Programmable Automation Controller (PAC) to record events or time stamping inputs in order to log the change of state for a point.
- An in-PAC chassis historian module logs historical data at Level 1 (skid/machine).
- High-speed applications can use timestamps to process inputs and outputs asynchronously from the control loop. For example, an application can use time-synchronized inputs and outputs to trigger a diverter without the application scan time matching the part cycle time.
CIP Sync uses IEEE 1588 Precision Time Protocol (PTP) to synchronize clocks in the control system. In the PTP architecture, all clocks are synchronized to a single grandmaster clock. In turn, this clock must be synchronized to Coordinated Universal Time (UTC) to represent the time of day in the system.
Several different approaches exist for setting time in the grandmaster clock that allows customers to meet different application requirements. For example, applications that only require rough correlation to UTC can use a handset grandmaster clock. In this case, the administrator simply sets the time in the grandmaster clock based on the time that is currently shown on another device such as their PC or smartphone. All clocks drift over time and need to be adjusted to accurately reflect UTC. Handset clocks do not have any inherent mechanism to compensate for drift. As such, the administrator needs to adjust the time on the grandmaster clock manually on a periodic basis to ensure that the clock is set to UTC.
Applications that require tight correlation to UTC can use a grandmaster clock with a built-in Global Navigation Satellite System (GNSS) receiver such as a Layer 3 IES or a dedicated PAC module (e.g., Allen-Bradley 1756-TIME module). These type of grandmaster clocks synchronize to the atomic clocks in the navigation satellites and automatically adjust their time to match UTC. However, the installation of these systems is complex and requires an antenna with an unobstructed view of the sky and low loss coaxial cable to connect to the receiver.
A final approach is the NTP-PTP flywheel feature available in some IES. This is a hybrid approach that uses Network Time Protocol (NTP) to synchronize and regulate the grandmaster clock to UTC. Correlation to UTC will not be as good as with a GNSS receiver; however, the flywheel does compensate for drift.
Figure 2 CPwE Time Architecture
In the CPwE Time solution (Figure 2), PTP time does not pass through the core and the PTP domain is bounded to the Cell/Area Zones connected to a single Layer 3 IES distribution pair. Within these bounds, the grandmaster clock will function as the reference for all PTP devices in the system. The preferred solution for the network topology is redundant star, however ring may work for some use cases. In either case, synchronization should be sufficient to support time stamping functionality of I/O modules that typically have a resolution of ±4μs to ±100μs. However, the accuracy of the timestamps to UTC will be limited by how well the grandmaster clock is correlated to UTC. Since PTP time does not synchronize across the core, correlation between PTP domains is also limited by how well the grandmaster in each of the PTP domains is correlated to UTC.
For more information on CIP Sync and CIP Sync applications, see Integrated Architecture and CIP Sync Configuration
https://literature.rockwellautomation.com/idc/groups/literature/documents/at/ia-at003_-en-p.pdf
Summary
CPwE is a collection of tested and validated architectures. The testing and validation follow the Cisco Validated Design (CVD) and Cisco Reference Design (CRD) methodologies.
The content of CPwE, which is relevant to both operational technology (OT) and informational technology (IT) disciplines, consists of documented architectures, best practices, guidance, and configuration settings to help industrial operations and OEMs with the design and deployment of a scalable, reliable, secure, and future-ready plant-wide industrial network infrastructure. CPwE can also help industrial operations and OEMs achieve cost reduction benefits using proven designs that can facilitate quicker deployment while helping to minimize risk in deploying new technology. CPwE is brought to market through an ecosystem consisting of Cisco, Panduit, and Rockwell Automation emergent from the strategic alliance between Cisco Systems and Rockwell Automation.
The Deploying Scalable Time Distribution within a Converged Plantwide Ethernet Architecture Design Guide (DG) outlines several use cases for designing, deploying and managing time synchronization technology throughout a plant-wide Industrial Automation and Control System (IACS) network infrastructure. CPwE Time Design Guide highlights the key IACS application requirements, technology, and supporting design considerations to help with the successful design and deployment of these specific use cases within the CPwE framework. CPwE Time was architected, tested and verified by Cisco Systems and Rockwell Automation with assistance by Panduit.
More information on CPwE Design and Implementation Guides can be found at the following URLs:
– http://www.rockwellautomation.com/global/products-technologies/network-technology/architectures.page?
– http://www.cisco.com/c/en/us/solutions/enterprise/design-zone-manufacturing/landing_ettf.html
Feedback