Distribution Automation Use Cases

Distribution Automation (DA) refers to the monitoring and control of devices located on the distribution feeders, such as line reclosers, load break switches, sectionalizers, capacitor banks and line regulators, and devices located in the distribution substation. DA is an overlay network deployed in parallel to the distribution feeder. It enables two-way communication between controllers used in the distribution feeder and the intelligence application that resides in the Utility control center or Secondary Substation for improving grid reliability, availability, and control. Figure 2 depicts a radial distribution feeder:

Figure 2 Distribution Feeder

In Figure 2, the distribution feeder can be observed coming out of the Secondary Substation; various distribution automation controllers (IEDs) in the feeder, such as the recloser controller, voltage regular controller, and capacitor bank controller, are positioned along the distribution feeder. Key functions and operations of Distribution Automation include protecting the distribution system, managing the fault, measuring the energy usage, managing the assets, and controlling and managing system performance. European feeders are largely three-phase and most European countries have a standard secondary voltage of 220, 230, or 240 V.

Use Cases

The following use cases of Distribution Automation will be discussed in this design guide:

■Volt/VAR Regulation

■Fault Location Isolation and Service Restoration (FLISR)

The radial feeder distribution system design is considered for Volt/VAR regulation use cases and the parallel feeder distribution system is considered for FLISR use cases. Cisco DA Gateways are very well suited for other feeder deployments such as mesh and loop distributed feeder designs.

Volt/VAR Control Use Cases and Benefits

This use case address automating dynamic and efficient delivery of power. Utilities look at achieving large saving by enhancing the efficiency of their power distribution infrastructure-in other words, improving the effectiveness of the flow of electricity. In order to evaluate the process, it is important to review the differences between what is called real power and reactive power.

■Real power is used to run all lights, devices and production lines. It is the power that "does the work."

■Reactive power does not contribute anything to doing work, but it does cause conductors to heat up and it takes up a certain amount of "space" in the wires.

The more reactive power flowing on a line, the less "room" there is for real power, and the less efficient is the distribution system.

Today, in order to eliminate or at least minimize reactive power flows, utilities have deployed on their local distribution systems devices, such as capacitor banks or special transformers that are typically located at substations or on the feeder. These devices work to keep reactive power flows down, making the full capacity of the conductor available for the real power. This process is known as Volt/VAR regulation or control:

■Power Factor Regulation/VAR Compensation—Improves efficiency of energy supply by ensuring voltage and current are in phase when supplied to the customer.

■Conservation Voltage Regulation—At times of peak load, ensure the minimum required voltage level is supplied to the customer.

■Volt/VAR Control—Power factor regulation + Conservation voltage regulation.

Volt/VAR Actors

Figure 3 depicts various actors used in the Volt/VAR use case. The actors used in the Volt/VAR use case are Load Tap Changers, Voltage Regulators, and Capacitor Bank Controllers (CBCs).

Figure 3 Volt/VAR Actors

Voltage Regulator and Load Tap Controllers

Voltage regulation functions are performed using the Voltage Regulator/Load Tap Controller actors. Voltage can be raised or lowered based on load conditions. Voltage Regulators are types of transformers that make small adjustments to voltage levels in response to changes in load. They are installed in substations (where they are called load tap changers) and along distribution feeders to regulate downstream voltage. Voltage Regulators have multiple "raise" and "lower" positions and can automatically adjust according to feeder configurations, loads, and device settings.

Capacitor Bank Controllers

CBCs are used to supply reactive power. Utilities use capacitors to compensate for reactive power requirements caused by inductive loads from customer equipment, transformers, or overhead lines. Compensating for reactive power reduces the total amount of power that needs to be provided by power plants, resulting in a flatter voltage profile along the feeder and less energy wasted from electrical losses in the feeder. A distribution capacitor bank consists of a group of capacitors connected together. Capacitor banks are mounted on substation structures, distribution poles, or are "pad-mounted" in enclosures.

Volt/VAR Application Communication Flow

In Figure 4, Volt/VAR and Supervisory Control and Data Acquisition (SCADA) applications are hosted in the DSO control center and RTU and load tap controllers are located in the Secondary Substation. The remote terminal unit (RTU) acts as an outstation device that proxies the poll and/or control command to various field devices like the CBC and end-of-line voltage monitor. This guide covers the use case scenario where the Volt/VAR application flow between the Intelligent Electronic Device (IED) and SCADA happens via RTU and the distribution feeder type considered is radial. A direct application flow from field devices to the control center for the Volt/VAR use case will be covered in future guides.

Figure 4 Volt/VAR Block Diagram

The detailed application flow between different actors for power factor regulation is depicted in Figure 5:

Figure 5 Power Factor Regulation Flows

1. Event class data poll to the following devices from RTU:

–Substation meter, poll measured Value (Short Floating Point) registers (0 to 4)

–All CBC(s), poll measured Value (Short Floating Point) (0) and double point command(0)

–End-of-line voltage monitor, poll measured Value (Short Floating Point) register (0)

2. The Volt/VAR Optimization processor processes the data received from the devices and makes a control command decision based on the power factor calculation.

3. The control command is sent to RTU via SCADA to CBCs to close the Capacitor Bank Controller N by writing in a Control Relay Output Block (CROB) command register in T104.

4. Event class data poll to the following devices from the RTU:

–Substation meter, poll measured Value (Short Floating Point) registers (0 to 4)

–All CBC(s), poll measured Value (Short Floating Point) (0) and double point command(0)

–End-of-line voltage monitor, poll measured Value (Short Floating Point) register(0)

5. All the above steps are repeated to all the CBCs on the feeder line to maintain a Power Factor value close to 1.

Figure 6 Conservation Voltage Regulation (CVR)

Figure 6 depicts the detail call flow involved in conservation voltage regulation.

1. Event class data poll to the below devices from RTU:

–Substation meter, poll measured Value (Short Floating Point) registers (0 to 4)

–All CBC(s), poll measured Value (Short Floating Point) (0) and double point command (0)

–End-of-Line voltage monitor, poll measured Value (Short Floating Point) register (0)

2. The Volt/VAR Optimization processor processes the data received from the devices and makes a control command decision based on the power factor calculation.

3. Control command is sent to RTU via SCADA to the load tap controller to lower/raise LTC by writing in a Control Relay Output Block (CROB) command register in T104.

4. Event class data polls to the following devices from RTU:

–Substation meter, poll measured Value (Short Floating Point) registers (0 to 4)

–All CBC(s), poll measured Value (Short Floating Point) (0) and double point command(0)

–End-of-Line voltage monitor, poll measured Value (Short Floating Point) register (0)

5. The above steps are repeated to maintain a Power Factor value close to 1 along the feeder line.

Fault, Location, Isolation, Service Restoration (FLISR)

FLISR Use Case and Benefits

FLISR is the process for dealing with fault conditions on the electrical grid. The following occurs as part of this process:

1. Detects (and locates) faults

2. Isolates the faults to the smallest segment of the grid possible

3. Restores as much service as possible while the fault is isolated

FLISR includes automatic sectionalizing and restoration and automatic circuit reconfiguration. These applications accomplish DA operations by coordinating operation of field devices, software, and dedicated communication networks in order to automatically determine the location of a fault and rapidly reconfigure the flow of electricity so that some or all of the customers can avoid experiencing outages. Because FLISR operations rely on rerouting power, they typically require feeder configurations that contain multiple paths to single or multiple other substations. This creates redundancies in the power supply for customers located downstream or upstream of a downed power line, fault, or other grid disturbance.

The benefits of FLISR include:

■Consumers experience minimal outage.

■Utilities improve the System Average Interruption Duration Index (SAIDI) and the System Average Interruption Frequency Index (SAIFI) numbers and avoid financial penalties being levied by the regulator.

FLISR application control can be implemented in the following modes:

■Supervised Mode-In supervised mode of operation, no automatic control, system delivers information to operator. Operator initiates manual control actions. Restoration time will be longer in this approach. Please refer to the Distribution Automation - Secondary Substation 1.0 Implementation Guide, which addresses this use case, at the following URL:

– https://salesconnect.cisco.com/#/search/Secondary%2520Substation%2520Implementation%2520Guide/conten t

■Semi Automatic Mode-A mix of automatic and supervised control is followed. The DA system automatically isolates the fault and performs the restoration part of upstream restoration. The upstream section is between the substation and the faulted section. Manual restoration operation is performed on the downstream section, which is between the fault section and the end of feeder. This guide will address this mode of operation. In this mode, communication happens between IEDs in field to the Distribution Management System (DMS) application residing in control center.

■Fully Automatic Mode-Isolation and restoration happens automatically without any dispatcher intervention. Communication happens directly between a group of associated IEDs. Restoration is very fast (<1 second), but this mode is a complex approach to deploy.

How FLISR Works

Figure 7 is divided into four parts (A,B,C, and D) to show how FLISR operations typically work.

■In Example A of Figure 7, the FLISR system locates the fault, typically using line sensors that monitor the flow of electricity, measures the magnitudes of fault currents, and communicates conditions to other devices and grid operators.

■Once located, FLISR opens switches on both sides of the fault: one immediately upstream and closer to the source of power supply (Example B of Figure 7), and one downstream and further away (Example C of Figure 7).

■The fault is now successfully isolated from the rest of the feeder. With the faulted portion of the feeder isolated, FLISR next closes the normally open tie switches to neighboring feeders. This re-energizes the unfaulted portion(s) of the feeder and restores services to all customers served by these unfaulted feeder sections from another substation/feeder (Example D of Figure 7).

Figure 7 How FLISR Works

FLISR Actors

■Recloser-The circuit recloser is a self-contained device with a necessary monitoring circuit to detect and interrupt over-current conditions and automatically reclose the line.

■Sectionalizing Switch or Remote Control Switch-Remote Controller Switches can be load break or fault interrupting devices.

■Remote Fault Indicator-Used to detect faults.

■Distribution Management System (DMS)-The DMS application residing in the DSO control center is an intelligent application, which is the brain of FLISR systems and which performs application circuit reconfiguration logic.

Figure 8 depicts a parallel feeder distribution system. Two distribution feeders are common out of two different Secondary Substations and each feeder has a recloser associated with it. Remote Fault Indicators and remote control switches are distributed across both feeders. RCS3 3 is, by default, an open switch.

Figure 8 FLISR Parallel Feeders

Figure 9 FLISR Application Communication Flow

In Figure 9, the application flow can be observed happening directly from feeder devices to the DMS application in the DSO control center. The flow is summarized below:

■Remote Fault Indicator (RFI) 1 reports to the Distribution Management System (DMS) whenever it encounters a fault.

■Recloser2 opens and send a report to DMS when it encounters a temporary fault.

■Recloser2 opens and send a report to DMS when it encounters a permanent fault.

■Remote Control Switch (RCS) 2 reports no voltage status to DMS.

■RCS 2 opens when it encounters faults for second time and send a report to DMS.

■DMS issues a close command to the RCS 3.

■DMS initiates a periodic poll (every minute) for the all feeder devices.

■DMS initiates a solicit periodic poll (every 5 minutes once) for all feeder devices.

Cisco Resilient Mesh and SEL FLISR Architecture

Cisco resilient mesh and SEL FLISR architecture are shown in Figure 10. SEL FLISR controller device deployment follows a centralized FLISR architecture. The SEL FLISR controller devices are the outstation devices. These outstation devices connect to Cisco IR510 DA gateways, enabling the communication path. Cisco CR mesh comprising IR510, IR530 can be aggregated at a CGR1000 series of routers, usually located in the substation/along the feeder. The CGR1000 series of routers can talk to the control center over Private WAN (or) Public Cellular and fiber networks. Cisco Resilient mesh provides communication infrastructure to the SEL FLISR Architecture.

Figure 10 Cisco Resilient Mesh and SEL FLISR Architecture

The SEL reclosers connect to the Cisco Resilient Mesh Industrial Routers (IR510) using Ethernet.

The Cisco IR510 DA gateways establish a reliable mesh wireless network based on signal propagation radio design. For instances where the radio signal cannot cover a certain area or signal levels are weak, Cisco Range Extenders (IR530) can be deployed to increase signal coverage for that area. CR mesh based on ISM 902-928MHz and IEEE802.15.4g/e standard using OFDM modulation with a physical data rate up to 1.2 Mbps can support the performance requirements of the FLISR application.

The SEL and Cisco DA solutions help utilities lower the SAIDI (System Average Interruption Duration Index) and SAFI (System Average Interruption Frequency Index) performance metrics which reflects the reliability of their Power Distribution network.

DNP3/IP messages between the SEL-651R recloser and the SEL controller are reliable routed through the mesh network via the most optimal path to the field area border router, such as Cisco CGR 1000 router, known as the mesh exit point, and then via either a fiber network or Cellular network to the Control Center location.

The SEL FLISR solution can also be deployed in a Centralized architecture where one or more RTAC devices are installed in the Control Center and each controller services an area that is not necessarily bonded to a substation service area. This approach has a lower deployment cost.

The Distribution Automation system consists of a DA controller (DAC) that communicates with multiple recloser controls, switch controls, feeder relays, and a wide range of other intelligent electronic devices (IEDs). The DAC is implemented on the SEL Real-Time Automation Controller (RTAC) family of controllers. The DAC can be applied to a wide variety of distribution network arrangements. The DACS addresses two major control objectives for the power distribution system: automatic reconfiguration (AR) and dynamic feeder optimization (DFO). This design will focus on the automatic reconfiguration since it has more stringent communications requirements.

The DAC functions to detect permanent fault and open-phase conditions on the distribution network. The DAC will act to isolate the affected zone of the feeder and restore power to the unaffected zones of the feeder from the normal source and from alternative sources, if available. The DAC also functions to detect overload conditions on the distribution network. The DAC will act to mitigate overloads by transferring load to adjacent feeders or by load shedding if alternative capacity is unavailable. In addition, the DAC functions to detect station events and loss-of-source events. The DAC will act to isolate the affected station and restore power to the unaffected feeders from alternative sources, if available.

Solution Overview

Cisco Resilient Mesh networks can support the SEL FLISR application over a variety of topologies and places in the network. Contents of the subsequent sections is summarized below:

These topologies were considered during the joint validation activity:

■FLISR Urban Topology

■FLISR Rural Topology

Each topology is described with the following:

■FLISR Topology-based one-line diagram (with segment division)

■FLISR Topology-based SEL device to Cisco device mapping diagram

■FLISR Topology-based SEL device to Cisco device mapping table.

FLISR Fault scenarios are described below:

For the SEL FLISR use case in Rural/Urban topology, the following faults in the power line are considered:

–Fault with Lock Out

–Open Phase Fault

–Loss of Source Fault

Each of these FLISR Fault scenarios are considered in three states:

–Normal state

–Fault state

–Restored or reconfigured state

SEL FLISR Use case Validation Call Flow Sequence

The call flow sequence of the SEL FLISR use case validation uses the DNP3/IP application protocol. UDP is the recommended protocol at the transport layer to transport the DNP3/IP messages. The two topologies for SEL FLISR use case validation are:

■FLISR Urban Topology

■FLISR Rural Topology

Figure 11 FLISR - Call Flow Event Sequence - Application DNP3 between DAC and IED

The Call flow sequence in Figure 11 is categorized into these major blocks:

■Class123 (Read/Response)

■Class0123 (Read/Response)

■Select/Response

■Operate/Response

■Unsolicited Response/Confirmation

Of the Commands listed above, Class123 is a periodic command; others are event-driven commands. For example, Unsolicited Response is an event-driven command.

Class0123 is used as periodic command, but with a longer duration of polling interval. The interval is once every 30 minutes in case of FLISR over radio.

In Figure 11, the Class123 event is configured to operate periodically every “N” seconds.

In the following FLISR Fault scenarios, these faults are introduced in specific segments of the Urban/Rural topology:

■Fault with LockOut

■Open Phase

■Loss of Source

When these faults are introduced into the SEL FLISR (Urban/Rural) topology into a particular segment, the SEL device in the affected segment raises “Unsolicited Response” messages to the SEL FLISR Controller (DAC), located in the control center.

DAC then sends an acknowledgment “Confirmation” message to the corresponding SEL device (Recloser Controller).

The “Unsolicited Response” conveys a change of state in the network. The SEL DA FLISR Controller performs “Class0123 Read” of all the related set of SEL devices to get the holistic view of the SEL FLISR Topology status.

In response to the Class0123 read request from the DAC, the SEL devices then respond with “Class 0123 Response” to the SEL FLISR Controller. The set of related SEL devices includes devices in the affected feeder section(s) and any adjacent sections connected with a normally open point.

The SEL FLISR Controller performs a FLISR computation, and decides which SEL recloser devices it must open, and which SEL recloser devices it must close. To accomplish this, the SEL FLISR Controller uses a “Select” message, followed by “Operate” message. Alternatively, the DA FLISR controller could directly perform “Operate” commands instead of Select before Operate. For each of the “Select” or “Operate” messages from the SEL FLISR Controller, the Recloser responds back with “Response” message.

Recommendation from SEL is to use direct Operate, as Select before Operate is a remnant of the past when auxiliary relays were used. This serves an additional purpose of reducing the load on the mesh by a fraction. Bandwidth conserved is the bandwidth earned for other application traffic.

When the status of the SEL device changes, the SEL recloser controller updates the SEL FLISR DA Controller device using “Unsolicited response” message, which is acknowledged back by DAC using “Confirmation” message.

When the SEL FLISR DA Controller requires a holistic view of the current status of the topology, it sends “Class0123 Read” to all the related SEL recloser controller devices. The current status of the related device is updated back in the corresponding “Class0123 Response” message sent by SEL recloser controller device to the DAC. The set of related SEL devices includes devices in the affected feeder section(s) and any adjacent sections connected with a normally open point.

This completes the SEL FLISR Event call flow sequence. This sequence could happen over SEL FLISR Urban/Rural topology, which are discussed in upcoming sections.

Note: Throughout this section, the terms DAC and SEL FLISR DA Controller are used interchangeably.

Cisco SEL FLISR Use case – Urban Topology

FLISR Topology, Urban Area - One-line diagram:

Figure 12 FLISR Topology, Urban Area – One-line diagram

The one-line diagram in Figure 12 describes the FLISR topology in urban area. The topology has been divided into ten segments (Seg1, Seg2,..., Seg10). Red boxes represent Energized (Closed) reclosers, while green boxes representing unenergized (Normally open) reclosers.

■Seg 1, Seg 2 and Seg 3 were energized from Transformer1

■Seg 4 was energized from Transformer2

■Seg 5, Seg 6, Seg 7 were energized from Transformer3

■Seg 8, Seg 9, Seg 10 were energized from Transformer4

■Reclosers 2, 4, 5 and 8 are in a Normally Open state

Urban FLISR Topology - SEL device to Cisco device mapping

Figure 13 FLISR Topology, Urban Area – SEL reclosers to Cisco device mapping

This topology captures the 1-to-1 mapping of SEL recloser devices to Cisco IR510 devices. The controller device is located in the Primary control center. CR Mesh is aggregated at the Field Area Network aggregator using CGR1000 series router which can be located in the substation. The communication between substation and control center can be over public/private WAN. The SEL device is positioned behind IR510 and connected using Ethernet.

In the above figure:

■The device representing DA CONTROLLER (SEL3530-2) is located in Primary control center.

■The device representing Recloser 9 (SEL3505-9) is associated with IR510-13.

■The device representing Recloser 6 (SEL3505-6) is associated with IR510-12.

■The device representing Recloser 1 (SEL3505-1) is associated with IR510-1.

■The device representing Recloser 10 (SEL505-10) is associated with IR510-23

■The device representing Recloser 7 (SEL505-7) is associated with IR510-22

■The device representing Recloser 2 (SEL505-2) is associated with IR510-21

■The device representing Recloser 8 (SEL505-8) is associated with IR510-33

■The device representing Recloser 4 (SEL505-4) is associated with IR510-32

■The device representing Recloser 5 (SEL505-5) is associated with IR510-42

■The device representing Recloser 3 (SEL505-3) is associated with IR510-4

The following table captures the Individual mapping of the SEL device with the Cisco Mesh and the mesh depth. The mapped pair of SEL/Cisco device is located on the mesh. The FLISR Controller (SEL3530-2) is located in the control center. The rest of the SEL devices (SEL 3505) are located along the substation and feeder.

Table 1 FLISR Urban Topology Components

The next section discusses a few different types of FLISR Faults (Fault with LockOut, Open Phase, Loss of Source), each of them in three different states (Normal, Fault, Restored).

FLISR Fault scenario - Fault with Lock Out

In the case of a permanent fault, the recloser goes into lockout state until the fault is resolved by utility technician. For example, when a tree branch falls on the distribution line and conductors break each recloser that noticed the fault will trip and send an unsolicited DNP3/IP message upstream to the DA FLISR Controller. This covers the Fault Location identification phase of FLISR.

When the fault location is identified, the DA controller gets the overall current status of the topology by sending a Class0123 poll to all the related reclosers. This group includes devices in the affected feeder section(s) and any adjacent sections connected with a normally open point. Then the DAC performs the FLISR restoration computation, and proceeds to the FLISR restoration phase.

In the FLISR restoration phase, the DA Controller sends DNP3/IP commands to the respective reclosers that needs to change state in order to restore the services. Once the DA FLISR controller finishes the reconfiguration, it performs one round of Class0123 polling to ensure the stability. This covers the Fault Isolation and Service Restoration phase of the FLISR.

Normal state

The Urban One-line topology in normal operational state is shown below.

Figure 14 Urban Topology - Fault with Lockout - Normal State

Fault state

When a fault is introduced in Segment 2 between Recloser1 and Recloser2, at point B, recloser1 on the segment changes its state from Normally Closed state (NC) to Open state. This results in loss of power to the segments shown in the figure below.

■Segment 2 (where the fault occurred)

■Segment 3 (where the fault did not occur)

Figure 15 Urban Topology - Fault with Lockout - Fault State

In the faulted state, customers in the region in segment 2 experience a loss of power; customers in the non-faulted region in segment 3 are also experiencing the loss of power.

The moment the fault occurred, recloser1 that detects the fault (in segment2) changes its state from Close to Open and enters Lockout state. The DA FLISR controller then sends Class0123 polling after one recloser goes into Lockout state, indicating there is a permanent fault.

Points to note in the topology:

■Recloser1: Should change the state from Normally Closed state to Open state and locked out.

■Recloser3: might continue to be in Closed state, but no power on the feeder line.

■Consumers of power located in Segment2 and Segment3 would experience loss of service.

Restored state

The FLISR SEL DA controller response sequence is outlined below:

1. To Change the state of the Recloser 3 located between segment 2 and 3 from Normally closed to Open

2. To change the state of the Recloser 5(located between segment 3 & 8) from Normally Open state to Closed state, provided transformer 4 doesn’t get overloaded to serve segment 3.

This results in energizing segment 3 with power source from transformer4 feeder.

Figure 16 Urban Topology - Fault with Lockout - Restored State

The results to customers is as follows.

■Customers in non-faulty region in segment 3 would have power service restored with the help of FLISR.

■Due to Fault isolation, power outage is restricted to Customers in the faulty (and isolated) region in segment 2.

Summary:

■Fault Location Identification - identified in segment2.

■Service Restoration – Restoring the power to non-faulty segment (segment 3).

■Isolation - Fault has been restricted to affected customers in segment 2 alone.

FLISR Fault scenario – Open Phase

An Open Phase fault applies to three-phase circuits, where one of the line voltage is lost. One cause could be due to a bad or loose street pole line jumper that interrupts the line. The loss of voltage will trigger a DNP3/IP unsolicited message to the DA Controller. The DA controller then initiates the necessary action, similar to FLISR Fault scenario - Fault with Lockout.

Normal state

Figure 17 FLISR Urban Topology – Open Phase - Normal State

Figure 17 portrays the normal state of the FLISR Urban topology. Take note of segment 8. Reclosers corresponding to this segment are Recloser 5, Recloser 8 and Recloser 10. Of these three reclosers, Reclosers 5 & 8 are Normally Open. Recloser 10 is in Closed state, thus energizing segment 8 with the power source from Transformer 4.

Fault state

The Open Phase Fault occurs in point B of recloser 10, as highlighted in below figure XX, with a red “X”.

Figure 18 FLISR Urban Topology – Open Phase - Fault State

Figure 18 portrays the fault state of the FLISR Urban topology. When fault occurred in segment 8 customers experience loss of power. The participating reclosers report the change of state using DNP3/IP Unsolicited Response message to the FLISR DA controller located in the control center.

The DA controller sends DNP3/IP Class0123 polling from all the related reclosers in the FLISR Urban topology to get a holistic view of the topology status. The set of related SEL devices includes devices in the affected feeder section(s) and any adjacent sections connected with a normally open point.

Note: In comparison to the FLISR Fault in the Fault with Lockout scenario, while in fault state only customers in the faulty segment 8 experience loss of power. All the other segments have power.

Restored state

Segment 8 is chosen to illustrate the following point. In cases where no other segment is affected, other than the faulty segment, the FLISR restoration action can be to do nothing.

Figure 19 FLISR Urban Topology – Open Phase - Restored State

In this state:

■The DA controller makes the decision to change the state of the Recloser10 from normally closed state to Open.

■Due to Fault isolation, power outage is restricted to customers in the faulty region in segment 8.

■No Customers in any other segment are affected.

Summary:

■Fault Location Identification - identified in segment8.

■Service Restoration – No FLISR restoration needed in this case. It is only the isolation of faulty segment.

■Isolation - Fault has been restricted to affected customers in segment 8 alone.

FLISR Fault scenario – Loss of Source

Loss of source applies to fault that occurs within the substation yard. For example, a bus fault at the substation could cause an outage for the entire feeder originating from that substation. In this scenario, the relay within the substation will notify the SEL DA controller of the loss of power which will then initiate the restoration process.

Normal state

Figure 20 Urban Topology – Loss of Source - Normal State

Figure 20 portrays the normal state of the FLISR Urban topology. Notice the feeder line from transformer 3 which includes segment 7, segment 6 and segment 5.

In normal operational state,

■Reclosers 4 & 8 are Normally Open. Circuit Breaker3, Reclosers 6 & 7 are Closed.

■All the three segments (5,6 and 7) derive a power source from transformer3.

Fault state:

The third feeder transformer was taken out of service to simulate a loss of power. The substation feeder breaker (Brk3) tripped and all the downstream customers lost power.

Figure 21 Urban Topology – Loss of Source - Fault State

In fault state:

■Reclosers 4 & 8 are still in Normally Open state.

■Circuit Breaker3, Reclosers 6 & 7 are in Closed state.

■All the three segments (5,6 and 7) experiences loss of power, as transformer3 is out of service.

The circuit breaker and recloser devices sends DNP3/IP unsolicited message (about the loss of power) upstream to the SEL DA controller device located in the control center. The SEL DA controller performs DNP3/IP Class0123 polling from all the related reclosers in the FLISR Urban topology to get a holistic view of the current state of the topology.

Restored state

The SEL FLISR DA Controller finds the most optimal way to restore the services and, in this case, it chooses to:

■Change the Circuit breaker 3 from Closed state to Open state.

■Change the Recloser8 state from Normal Open to Closed state.

This would result in energizing segments (5,6 and 7) with power source from transformer4 feeder.

Figure 22 Urban Topology – Loss of Source - Restored State

In this restored state:

■Customers in all the affected segments (5,6 and 7) would have power service restored with the help of FLISR.

■Due to Fault isolation, transformer3 is taken out of the picture. Affected segments are now served by power from transformer4.

Summary:

■Fault Location - identified fault in transformer3.

■Isolation - Fault has been restricted to transformer3 and it has been taken out of service.

■Service Restoration – Restored the power to affected segments using alternate power source (transformer4).

Cisco SEL FLISR Use case – Rural Topology

FLISR Topology – Rural Area - One-line diagram:

Figure 23 FLISR Topology (Rural Area) – One-line diagram

Figure 23 captures the one-line diagram of the FLISR topology in a Rural area. The topology has been divided into eleven segments (Seg1, Seg2,... Seg11). Red boxes represent Energized (Closed) power line, and green boxes represent unenergized (open) power line. Segments are energized as follows:

■Seg 1-6 are energized from Transformer1

■Seg 7-11 are energized from Transformer2

Rural FLISR Topology - SEL device to Cisco device mapping:

Figure 24 FLISR Topology (Rural Area) – SEL reclosers to Cisco device mapping diagram

Figure 24 captures the 1-to-1 mapping of SEL recloser devices to Cisco IR510 devices. The DA controller device is located in the Primary control center. CR Mesh is aggregated at the Field Area Network aggregator using CGR1000 series router which can be located in the substation. The communication between substation and control center can be over public/private WAN. The SEL device is positioned behind IR510 and connected using Ethernet.

The following table captures the Individual mapping of the SEL device with Cisco Mesh device and the mesh depth. The mapped pair of SEL/Cisco devices is located on the mesh. FLISR Controller (SEL3530-2) is located in control center. Other SEL devices, SEL 3505s, are located along the substation and feeder.

Table 2 FLISR Rural Topology Components

A few different types of FLISR Faults such as Fault with LockOut, Open Phase, Loss of Source, each of them in three different states (Normal, Fault, Restored) are discussed.

SEL FLISR Rural Topology - FLISR Fault scenario - Fault with LockOut

This failure scenario in Rural topology is very much similar to “SEL FLISR Urban Topology - FLISR Fault scenario - Fault with LockOut”. From the SEL FLISR Use case point of view, there is not much difference. Affected segments sends DNP3/IP unsolicited messages to SEL FLISR DA Controller located in the control center, which in turns performs Class0123 polling to know the holistic view of the FLISR topology. Later, SEL DA controller performs the required restoration operations. However, the underlying topology of the Cisco RF mesh is different. Rural topology uses hierarchical multi-hop topology.

Normal state

The Rural One-line topology in normal operational state looks like Figure 25 below.

Figure 25 Rural Topology - Fault with Lockout - Normal State

In above Figure 25:

■Recloser 6 is in Normally Open (NO) state. Every other device is in closed state (the line is energized).

■Segments 1-6 are powered by feeder from transformer1

■Segments 7-11 are powered by feeder from transformer2

Fault state:

When a fault gets introduced in Segment 3 (between Recloser2 and Recloser3), recloser 2 participating in that segment changes its state from Normally Closed state (NC) to Open state. This would result in loss of power to below segments (as shown in the figure below):

■Segment 3 (between Recloser 2 and 3) - where the fault did occur

■Segments 4-6 (between Recloser 3 and 6) - where the fault did not occur.

Figure 26 Rural Topology - Fault with Lockout - Fault State

In this state, customers in the faulty region in segment 3 is experiencing loss of power. Along with them, customers in non-faulty regions belonging to segments 4-6 are also experiencing the loss of power.

The moment fault occurred, recloser2 participating in the segment (3) changes its state from Closed to Open. Meanwhile, following reclosers would send DNP3/IP unsolicited messages upstream to the SEL FLISR DA controller:

■Recloser2 – conveys the fault.

■Reclosers (3-6) – conveys the loss of power.

The SEL FLISR DA controller then performs Class0123 polling on every related recloser to get holistic view of the topology before making any restoration decision.

Restored state

Figure 27 Rural Topology - Fault with Lockout - Restored State

The FLISR SEL DA controller decides:

1. To change the state of Recloser 3 (located between segment 3 & 4) from normally closed state to Open state.

2. To change the state of Recloser 6 (located between segment 6 & 7) from Normally Open state to Closed state. This would result in energizing segments (4-6) with power source from transformer2 feeder.

In this state:

■Customers in non-faulty region in segments 4-6 would have power service restored with the help of FLISR.

■Due to Fault isolation, power outage is restricted to Customers in the faulty region in segment 3.

Summary:

■Fault Location Identification - identified in segment3.

■Service Restoration – Restoring the power to non-faulty segment (segment 4-6).

■Isolation - Fault has been restricted to affected customers in segment 3 alone.

SEL FLISR Rural Topology - FLISR Fault scenario – Open Phase

Open Phase fault applies to three phase circuits, where one of the line voltage is lost. One cause could be due to a bad or loose street pole line jumper that interrupts the line. The loss of voltage will again trigger a DNP3/IP unsolicited message to the DA Controller. DA controller then initiates the necessary action, similar to FLISR Fault scenario - Fault with Lockout.

Normal state

Figure 28 FLISR Rural Topology – Open Phase - Normal State

Figure 28 portrays the normal state of the FLISR Rural topology. The emphasis is placed on segment 8 and segment 7. Reclosers corresponding to these segments are Recloser 6,7 and 8.

In above Figure 28:

■Recloser 6 is in Normally Open state. Every other device is in closed state (means, line is energized).

■Segments 1-6 are powered by feeder from transformer1

■Segments 7-11 are powered by feeder from transformer2

Fault state

The Open Phase Fault occurs in point A of recloser 7, as highlighted in below figure XX, with a red “X”.

Figure 29 FLISR Rural Topology – Open Phase - Fault State

Figure 29 portrays the fault state of the FLISR Rural topology, when fault occurred on segment 8.

This would result in loss of power to below segments:

■Segment 8 (between Recloser 8 and 7) - where the fault did occur

■Segment 7 (between Recloser 7 and 6) - where the fault did not occur.

In this state, customers in the faulty segment (8) is experiencing loss of power. Along with them, customers in non-faulty segment (7) is also experiencing the loss of power. Before the fault occurred, both of these segments (7 & 8) used to receive power from transformer2.

The moment fault occurred on segment 8, the participating reclosers would be sending DNP3/IP unsolicited messages upstream to the SEL FLISR DA Controller located in the control center. The SEL FLISR DA controller then performs Class0123 polling on every related recloser in the FLISR Rural topology, to get a holistic view of the topology before making any restoration decision.

Restored state

Figure 30 FLISR Rural Topology – Open Phase - Restored State

The FLISR SEL DA controller decides:

1. To change the state of the Recloser 8 (located between segment 9 & 8) from Closed state to Open state

2. To change the state of the Recloser 7 (located between segment 8 & 7) from Closed state to Open state

3. To change the state of Recloser 6 (located between segment 6 & 7) from Normally Open state to Closed state. This would result in energizing segment (7) with power source from transformer1 feeder.

In this state:

■Customers in non-faulty region in segments 7 would have power service restored with the help of FLISR.

■Due to Fault isolation, power outage is restricted to Customers in the faulty region (segment 8).

Summary:

■Fault Location Identification - identified in segment8.

■Service Restoration – Restoring the power to non-faulty segment (segment 7).

■Isolation - Fault has been restricted to affected customers in segment 8 alone.

SEL FLISR Rural Topology - FLISR Fault scenario – Loss of Source:

Loss of source applies to fault that occur within the substation yard. For example, a transformer going bad (or) bus fault in the substation, that causes an outage for the entire feeder originating from that substation. In this scenario, the relay within the substation will notify the SEL DA controller of the loss of power which will then initiate the restoration process.

Normal state

Figure 31 Rural Topology – Loss of Source - Normal State

Figure 31 portrays the normal state of the FLISR Rural topology. The emphasis is placed on entire feeder line from transformer 1 (which include segments 1-6) In normal operational state,

■Recloser 6 is in Normally Open state. All other devices Circuit Breakers & Reclosers are in Closed state.

■Segments 1-6 deriving power from transformer1

■Segments 7-11 deriving power from transformer2

Fault state

The first feeder transformer was taken out of service to simulate a loss of source. Once the transformer1 is taken out of service, all the downstream customers in segments (1-6) would experience loss of power.

Figure 32 Rural Topology – Loss of Source - Fault State

In fault state:

■Segments 1-6 would experience loss of power, as transformer1 is out of service.

■Reclosers 6 is still in Normally Open state.

The circuit breaker and recloser devices sends DNP3/IP unsolicited message upstream to the SEL FLISR DA controller device located in the control center, about the loss of power in the feeder line. The SEL DA controller performs DNP3/IP Class0123 polling from all the related reclosers in the FLISR Rural topology to get a holistic view of the current state of the topology.

Restored state

The SEL FLISR DA Controller finds the most optimal way to restore the services and, in this case, it chooses to:

■Change the Circuit breaker 1 from Closed state to Open state.

■Change the Recloser6 state from Normal Open state to Close state, provided the additional load will not cause an overload on transformer2.

■This would result in energizing segments (1-6) with power source from transformer2 feeder.

Figure 33 Rural Topology – Loss of Source - Restored State

In this state:

■Customers in all the affected segments (1-6) would have power service restored with the help of FLISR.

■Due to Fault isolation, transformer1 is isolated by opening Circuit Breaker1. Affected segments are now served by transformer2.

Summary:

■Fault Location Identification - identified in transformer1.

■Service Restoration – Restored the power to affected segments using alternate power source (transformer2).

■Isolation - Fault has been restricted between transformer1 and breaker1.

Evaluating Number of DA devices in mesh for given FLISR Application in single PAN – A Methodology

Figure 34 Methodology for evaluating number of DA devices in mesh for FLISR Application

Can a given FLISR application be run over CR mesh? The answer to that question lies in understanding the characteristics of the given FLISR application in terms of payload sizes exchanged, and their respective weightage. Weightage in this case is the number of occurrences of a particular payload size in FLISR packet profiling. Based on the understanding of the FLISR application traffic profile, FLISR mix of payload sizes could be formed. It is then recommended to run performance and latency tests for the FLISR mix of payload sizes, across various Ranks.

Obtained throughput and latency could then be evaluated to check if it could satisfy the FLISR application bandwidth and latency requirement. Combining this observation, along with the maximum accepted duration for the completion of FLISR application use case, the possible number of DA devices that could be supported in the CR mesh could possibly be derived.

Table 3 FLISR Mix of Payload sizes for 10 participating devices – A sample

Note: 372 Datagrams in above table corresponds to Application goodput in sample FLISR use case. All the discussion below revolves around goodput to derive the number of traffic generating DA (utility) devices in the CR mesh.

Common Design Considerations

This section covers the common design considerations, various stages involved in it, and different types of traffic that consumes/competes for the available mesh capacity.

Figure 35 CR Mesh Design Considerations

RF Planning

Before actual RF mesh deployment, theoretical planning of the RF site needs to be evaluated with the help of RF planning tools like ATDI. Details like radio and antenna parameters are entered into the RF planning tool. Locations for IR510, IR530 and CGR also needs to be fed into the tool.

The objective of this RF planning exercise is to predict the received signal strength of any given RF link.

RF Site Survey

With the idea derived from the RF planning tool, the theoretical data must be verified with the help of live field tests. In this phase, the CR mesh devices like IR510, IR530 and CGR would be deployed in the planned locations, along with antennas mounted in desired height. Check the received and forward signal strength of any given RF link. Ensure, the RSSI values measured during this RF site survey phase for any particular RF device to be in the expected range, as planned during the RF planning phase.

In this phase, presence of any interference would also be considered. Interferences could be temporary or permanent interference. Also, the noise floor be evaluated with the help of spectrum analyzer.

Note: If the in-band interference is affecting the performance numbers of the node, you may consider improving the quality of the links by adding a range extender. To main stronger signal strength, consider RSSI range greater than or equal to -80 dBm.

For out of band interference, consider using bandpass filter.

Interference could be of transient nature or of permanent nature.

■Transient interference conditions could be due to weather changes like RF foliage, obstacles in front of the antennas, or metallic objects in the way that change after a short time. In these instances, the mesh would self-heal and would retry transmission across all channels. When the conditions improve the performance returns to normal. The retry mechanism would resolve the issues as conditions improve.

■Permanent interference: As part of RF site survey, if the mean reading indicates that certain sections of the ISM band is excessively used, then operator may consider notching the selective channels.

Note: CR Mesh performance would be proportional to the number of channels being used.

RF Radio medium

RF radio is a half-duplex medium. The radio could either transmit or receive at any point in time. The IR510, IR530 and CGR could support OFDM, 2FSK modulations. In OFDM, 5 data rates are supported (1200, 800, 400, 200 and 50 Kbps). Each of these modulations have a different data rate and RSSI budget. For more details of the data rate supported, please refer to the R510 datasheet.

Data Rate vs Packet Rate vs Goodput

Theoretical maximum throughput for MAC/PHY technologies are expressed in Kbps, Mbps. Meanwhile, one measurement metric is packet rate, which signifies the number of packets that could be transmitted in any given second. Packet rate would vary according to the packet size under consideration. For example, comparing individual packet sizes of 64 vs 512, packet rate for 64 would be more than packet rate of 512. For a range of packet sizes, a packet mix could be composed, and the maximum packet rate could be obtained for the chosen packet mix.

Figure 36 Max Data Rate vs Packet rate vs Goodput

Maximum data rate represents the physical rate of the modulation/PHY Technology chosen (for example, OFDM 800). For a chosen modulation and maximum data rate, and for the chosen packet size (or mix of packet sizes), (packet rate) number of packets could be transmitted. Of these packet rate, some packets could get dropped due to congestion, and may have to be retransmitted. Since the retransmission also has to use the same physical capacity, net effective capacity that is available for application traffic, discounting the retransmission and lower layer communication overheads could be referred as Goodput, in our context.

The key behind extracting the maximum performance out of the CR mesh lies in designing the network, by minimizing the retransmission, and by reducing the possibilities of congestion. It is also vital to clearly understand the characteristics of the application traffic that the mesh needs to carry and consider it while designing the network.

Note: One important recommendation is to plan the CR mesh for 40-50% utilization of its full capacity, leaving the rest for peak traffic, and with some room for future growth.

It is not good practice to plan the CR mesh for 90-100% utilization of its capacity, which is inviting for instability considering the physical nature of the congestion and retransmission in half duplex medium.

Also, UDP as a layer4 transport protocol would offer better performance when compared to TCP in CR mesh, and hence usage of UDP is highly recommended.

Also, it is recommended to configure RPL (Routing Protocol for Low-Power and Lossy Networks) to operate in storing mode, which would eliminate the source routing requirement for the downstream communication to the lower rank devices. This reduces the “source routing header” overhead which in turn could improve the mesh performance. Storing mode when enabled, has an additional benefit to allow peer to peer traffic inside the PAN.

It is highly recommended to mark any latency sensitive critical traffic with dscp value of 18 (AF21), and the next level of moderately critical traffic with dscp value of 10 (AF11). Every other traffic could either be marked with dscp value of 0 or could be left unmarked. Ideal way is to have the QoS dscp values of the packet marked by the Utility controller devices located in the field, as well as by the devices located in the control center. For more details, please refer to the Quality of Service section.

CR Mesh Capacity Planning

Following are some of the different types of traffic that consumes/competes for the available mesh capacity.

■CR Mesh control plane traffic (RPL routing, keepalives, multicast control plane, and so on).

■CR Mesh devices (IR510, IR530) onboarding traffic (IEEE 802.1x, IEEE 802.11i, DHCPv6, and so on).

■CR Mesh devices management traffic (CoAP, CSMP, and so on).

■CR Mesh devices troubleshooting traffic (ping, traceroute, and so on).

■CR Mesh devices Upgrade and other necessity driven traffic (Firmware upgrade, IOx upgrade, security update, and so on).

■Application traffic – periodic (Example, periodic DNP3 Class123 polling every 5 minutes)

■Application traffic – event driven (Example, power outage, fault identification, and successive poll/control traffic).

■Different types of protocols used for Application communication – (Example, DNP3, MODBUS, IEC 60870, DLMS/COSEM, and so on.)

■Utility device management traffic (Example, Accessing Utility controller device behind IR510)

Figure 37 Mesh Capacity Planning

Figure 37 captures a sample mesh capacity allocation for different types of traffic that competes for bandwidth.

Considering the mesh capacity available between CGR and all Rank1 IR510/IR530 as 100%, plan 50% of it for the NMS and application use cases. Leave the remaining 50% as buffer for upgrades and other necessity driven traffic, and for future scaling of number of mesh devices. Of the planned 50% of mesh capacity, 10% could be assumed for control plane traffic as well as NMS related operations like (mesh device onboarding, management and troubleshooting). 40% of the mesh capacity could be planned for Application use case, which includes both periodic traffic, as well as event driven traffic.

Notes:

While the recommendation for planned capacity is 50%, it could be increased up to a maximum of 70% with 30% headroom.

The numbers referenced in the below section is for demonstration purpose only, just to drive home the procedure. The actual performance and latency numbers for the given application needs to be derived, as mentioned in the sub section Evaluating Number of DA devices in mesh for given FLISR Application in single PAN – A Methodology.

While planning for the worst case and considering the weakest, obtained result could be better than expected.

The Goodput obtained during the performance and latency tests for the given FLISR mix of payload sizes could serve as a good reference for the mesh capacity. For example, if Goodput obtained at Rank1 is 70 datagrams per second for FLISR mix of payload sizes, these 70 datagrams/s could be referred as 100% mesh capacity for all Rank1 nodes. Planning for 40% of the mesh capacity at Rank1 translates to 40% (70) = 28 Datagrams per second (DPS) for entire Rank1.

This 40% capacity (i.e., 28 Datagrams per second) at Rank1 could then be planned for sharing with lower rank nodes. If the FLISR application use case requires 372 datagrams (for 10 devices) to get completed, then 10 devices positioned at Rank1 could take (372/28=) ~14 seconds worth of mesh capacity at 40% loading to complete the FLISR use case. In reality, this could get completed faster, as there might be unused mesh capacity from the remaining 60%. Again, this is a sample number just to demonstrate the procedure.

In case of peer to peer traffic, there could be multiple simultaneous transmissions in the mesh at different hops and channels. This adds extra capacity to the mesh that is not considered in above explanation.

Determining the number of DA devices and mesh depth in the CR mesh for given application:

The depth or the number of hops/ranks in the CR mesh could be dictated by the latency requirement of the use case application.

Note: The next section assumes a few numbers to demonstrate the concept. These numbers do not represent the actual performance or latency numbers of the CR mesh. Run the performance and latency tests to arrive at the actual numbers pertaining to your CR mesh deployment.

Determining the mesh depth

To demonstrate an example, certain assumptions are made and the procedure to determine the mesh depth is shown

Assumptions:

■Application requirement: Mesh needs to cater to an application that requires 100ms of one-way latency.

■The latency requirement of 100ms is between CR mesh device (IR510) and control center.

■Per-hop latency based on the test results, if found to be in the range of 10-20 ms.

Assuming the worst-case latency of 20ms for one hop, 5 hops might require 100ms of latency. Given such data points, it is recommended to retain the mesh depth to 5 hops or less.

Similarly, if the application is fine with 1 second of one-way latency, theoretically the derivation could be something like (1000ms/20ms=) 50 hops. However, it is recommended not to exceed 8-10 rank depth, especially when the CR mesh is also used for latency sensitive traffic.

Note: Although this section recommends 8-10 rank depth maximum, if the requirement is to reach remote areas with poor connectivity, you could consider increasing the hop depth with range extenders. Choosing to operate at a lower data rate (For example, OFDM 50 Kbps) along with increased hop depth is another option to consider.

Determining the number of DA devices In the CR mesh

As a pre-requisite for this section, performance test must be run for the given application packet profile, as mentioned in the section “Evaluating Number of DA devices in mesh for given FLISR Application – A Methodology”, and obtain the “GOODPUT” measured in Datagrams per second (DPS) for the given Application mix of payload sizes (example, FLISR payload mix).

This section considers only 40% of mesh GOODPUT for the given application at any rank(hop), as per the recommended design. In other words, the intended application load could go up to a max of 40% of mesh capacity plus retransmissions. Meanwhile, at any given second, the then remaining and unused capacity would also be available for application to use, for even better performance.

Note: Data in below table considers the GOODPUT between mesh nodes and control center, without considering the extra capacity obtained when running peer-to-peer traffic.

To obtain the below table output, Bidirectional Multiple flow iperf3 tests needs to be executed with condensed version of multiple payload sizes and its respective weightage as referred in Table 3.

Table 4 Mesh Capacity planning – GOODPUT Allocation table – A sample

Notes:

The numbers shown in the above table are only sample values, chosen for the purpose of driving the methodology used to derive the number of DA devices in the mesh. Actual numbers could vary depending on the chosen Application packet profiling, and its corresponding performance and latency numbers over the CR mesh.

Data in above Table 4 assumes OFDM modulation with data rate of 800 Kbps. The numbers would vary according to the chosen modulation and data rate.

The 100% Capacity assumed in the above table refers to the multiple node throughput at the same rank. For example, 70 DPS in above table refers to cumulative throughput of multiple nodes in Rank1. If only one node is present in Rank1, 100% capacity could be 2X times approximately (for example,2 x 70 DPS = 140 DPS). Only multiple node throughput has been considered throughout the calculation.

The data in above table is for data originating devices only (devices generating utility goodput traffic). Inclusion of range extenders anywhere in the path doesn’t affect the calculation of number of DA devices.

For example, in case of three rank hierarchy (Rank1-3), if the rank3 device does not have good communication link with rank2 device, inclusion of range extender between rank2 and rank3 device would still be fine, as range extenders only relays the existing traffic, and doesn’t introduce any new data traffic. Hence, a mesh hierarchy with 3 IR510 and one IR530 should be considered as Rank 1-3 with respect to above Goodput allocation table.

Similarly, a mesh hierarchy with 5 IR510 and 2 IR530 could be considered under Rank1-5 category with respect to above goodput allocation table.

Data points required to derive the number of DA devices, for given application use case:

■Goodput DPS for all devices in Rank1. (Example, if 8 nodes are in Rank1, obtain cumulative DPS for all 8 nodes). This corresponds to number 28 in above table.

■Goodput DPS for 3 node hierarchy under any Rank1. This corresponds to number 14 in above table for Rank1-3 hierarchy.

■Goodput DPS for 5 node hierarchy under any Rank1. This corresponds to number 9.4 in above table for Rank1-5 hierarchy.

■Goodput DPS for 10 node hierarchy under any Rank1. This corresponds to number 6.8 in above table for Rank1-10 hierarchy.

Data point taken based on FLISR Application packet profiling for 10 devices:

■Total number of datagrams required for one FLISR application use case to complete (with 10 devices) = 372 datagrams.

Below section shall be discussed as multiple cases. The mesh could contain multiple rank1 devices. The term “10 nodes per rank1 hierarchy”, refers to linear mesh of 9 CR mesh devices, under one CR mesh rank1 device (IR510/IR530). Similarly, the term “3 nodes per rank1 hierarchy” would refer to linear mesh of 2 CR mesh devices under one rank1 device.

The application use case considered for below case study is FLISR, and it took 372 datagrams for FLISR to complete for a total of 10 participating devices. This would be used as reference for completion of FLISR use case/10 nodes, in the below cases.

Table 5 Number of DA devices in mesh – summary table

Table 5 captures the summary of recommendations given for different scenarios. For more details, refer to below sections.

Figure 38 Number of DA devices in CR mesh – Two Phases

■Number of devices derived based on CGR WPAN goodput rate. This is scaling to fit within WPAN capacity at any given second. Example: 7 devices with goodput rate of 4 datagrams per second (DPS) is catered by WPAN which has goodput rate of 28 DPS. Therefore, 7 devices could be allowed to transmit at any given second.

■Considering some time buffer, arrive at Benchmark unit for further time scaling.

■Time Scaling: Number of devices derived based on Application acceptable maximum duration limit. This is scaling to fit the number of DA devices over period of time. For example, if total datagrams for use case to complete is 72, 7 devices (each with a goodput rate of 4 DPS), would take (72 / (7*4) =) 3 seconds for use case to complete.

–Assuming 3X time buffer, consider 12 seconds instead of 3 seconds for one use case.

–Time taken for 1 set of 7 devices = 12 seconds.

–This way, if application accepted maximum duration limit = 120 seconds, then the mesh could cater up to a maximum of 10 sets (120s/12s per set of 7 devices). Theoretically, these 10 sets with 7 devices each could go up to a maximum of 70 devices.

■The resulting capacity scaling count = 7 devices; Time scaling count = 70 devices.

■Which means that 70 is the number of DA devices that could be deployed over CR mesh for the given application traffic characteristics.

Case1: Considering all nodes in Rank1.

This case assumes all the devices are positioned in Rank1 level itself.

Data point assumption:

■Goodput rate at rank1 = 28

With goodput of 28 DPS, 372 datagrams for FLISR application could take (372/28=) 14 seconds. In fact, it could be lesser than 14 seconds considering that application traffic is planned only for 40% of the mesh capacity, and any portion of the remaining 60% could possibly be available, thus having a possibility to offer a performance, better than 14 seconds. However, below calculation considers 14 seconds as per the 40% design recommendation.

The 372 datagrams FLISR packet profiling was based on 10 devices. So, 10 rank1 devices could take approximately 14-15 seconds to complete the FLISR use case.

Table 6 Number of DA devices – All Rank1 nodes - scaling type and considerations

Above table is a theoretical extrapolation. In practical, it is advised to reduce the number of devices that competes for the same transmission medium.

The number of devices that could be positioned in the mesh depends on the accepted maximum duration for FLISR use case. If the FLISR use case accepted maximum duration is 2-3 minutes, consider positioning 30-40 Rank1 nodes. If the FLISR use case accepted maximum duration is 4-5 minutes, consider positioning 45-60 Rank1 nodes.

Case2: Considering up to 10 nodes per rank1 hierarchy

This case assumes the Application latency requirement permits 10 hops of IR510/IR530 to be positioned under any Rank1 device.

Data point assumptions:

■Goodput DPS for Rank1-10 nodes = 6.8.

■Up to 10 nodes are considered per rank1 hierarchy.

■Goodput rate at rank1 = 28.

With goodput of 6.8 (~7) DPS, 372 datagrams for FLISR application could take (372/6.8=) 55 seconds.

At the rate of 7 DPS per rank1, and with goodput rate of 28 DPS for all Rank1, (28/7=) 4 Rank1 devices and its 10-node hierarchy should be able to transmit comfortably, without much of retransmission. This translates to 4 rank1 * 10 devices per rank1= 40 devices.

This could be fine, when the accepted duration for FLISR use case to complete is 2 minutes or above (considering 2 x 55 seconds).

Figure 39 Considering up to 10 nodes per rank1 hierarchy

If the accepted maximum duration for FLISR to complete is 4-5 minutes, we could afford to have some congestion and retransmissions at the Rank1. Hence, we could add more branches and increase the number of DA devices positioned under the CR mesh. For example, if 2-3 minutes could serve 4 Rank1 capacity, allowing some buffer for retransmission, 4-5 minutes could serve 5-6 Rank1 devices worth of capacity, each with hierarchy of nodes underneath it.

Table 7 Number of DA devices – up to 10 nodes per rank1 hierarchy - scaling type and considerations

Case3: Considering up to 5 nodes per rank1 hierarchy

This case assumes the Application latency requirement permits 5 hops of IR510/IR530 to be positioned in any Rank1 hierarchy.

Data point assumptions:

■Goodput DPS for Rank1-5 nodes = 9.4

■Up to 5 nodes are considered per rank1.

■Goodput rate at rank1 = 28.

■Considered FLISR packet profiling has 372 datagrams for 10 devices.

Figure 40 Considering up to 5 nodes per rank1 hierarchy

All the calculations below assume only 40% of full capacity, factoring in multiple nodes in same rank. For example, 28 DPS in Rank1 is the cumulative of all rank1 nodes, at 40% capacity.

With goodput of 9.4 DPS per branch of 5 nodes, combined goodput for 10 nodes = 18.8 DPS

Time taken to complete FLISR application/10 nodes = (372/18.8) = 20 seconds.

CGR WPAN (at 40% capacity) could serve FLISR application traffic worth 15 nodes at any point in time. (example, 9.4 DPS * 3 = 28.2 DPS, close enough to 28 DPS offered by Rank1).

In 20 seconds, 15 nodes worth of FLISR application traffic could be served.

This scaling limit comes from the CGR WPAN per second capability (again at 40% planned capacity). In reality, any unused/available mesh capacity of the remaining 60% would contribute to even better performance.

Another level of device scaling could be achieved based on maximum accepted duration for the completion of FLISR use case (for example, it could be 1 minute or 2 minutes or 5 minutes or N minutes). To achieve that, consider some time buffer and arrive at a benchmark unit of measurement. For example, at the rate of 15 nodes/20 seconds, theoretically 45 nodes could be served in 1 minute. However, allowing some time buffer, only 30 nodes are considered for 2 minute interval, and only 15-30 nodes were considered for 1 minute interval.

Table 8 Number of DA devices – up to 5 nodes per rank1 hierarchy - scaling type and considerations

As per above table, the number of devices that could be positioned in the mesh depends on the accepted maximum duration for FLISR use case. If the FLISR use case accepted maximum duration is 2 minutes, consider positioning 30 devices. If the FLISR use case accepted maximum duration is 4-5 minutes, consider positioning 50-60 devices (with 12 Rank1 and with 4 devices under each Rank1 node). The above design considers mesh with sufficient time buffer to deliver a better FLISR performance.

Note: The additional number of mesh devices can be positioned on the mesh, considering the buffer that was accounted in beginning. However, it is recommended to monitor the network for its expected performance, before adding additional number of devices, in incremental stages.

Case4: Considering up to 3 nodes per rank1 hierarchy

This case assumes the Application latency requirement permits 3 hops of IR510/IR530 to be positioned in any Rank1 hierarchy.

Data point assumptions:

■Goodput DPS for Rank1-3 nodes = 14

■Up to 3 nodes are considered per rank1.

■Goodput rate at rank1 = 28.

■Considered FLISR packet profiling has 372 datagrams for 10 devices.

Figure 41 Considering up to 3 nodes per rank1 hierarchy

All the calculations below assume only 40% of full capacity, factoring in multiple nodes in same rank. For example, 28 DPS in Rank1 is the cumulative of all rank1 nodes, at 40% capacity.

At the rate of 14 DPS per rank1, and with goodput rate of 28 DPS for all Rank1, (28/14=) 2 Rank1 devices and its 3-node hierarchy should be able to transmit comfortably, at full application planned capacity. This translates to positioning 2 rank1 * 3 devices per rank1= 6 devices without much of congestion and retransmission.

The reference 372 datagrams was profiled for 10 devices. This translates to 224 datagrams for 6 devices.

With goodput of 28 DPS for 6 devices, 224 datagrams for FLISR application could take (224/28=) 8 seconds for 6 devices. In 16-20 seconds, goodput from 12 devices could be served. In 24-30 seconds, goodput from 18-20 devices could be served.

Another level of device scaling could be achieved based on maximum accepted duration for the completion of FLISR use case (for example, it could be 1 minute or 2 minutes or 5 minutes or N minutes). To achieve that, consider some time buffer and arrive at a benchmark unit of measurement. For example, at the rate of 18 nodes/30 seconds, theoretically 72 nodes could be served in 2 minutes. However, allowing some time buffer, only 30 nodes are considered for 2-minute interval., and only 20-30 nodes were considered for 1-minute interval.

Table 9 Number of DA devices – up to 3 nodes per rank1 hierarchy - scaling type and considerations

As per above table, the number of devices that could be positioned in the mesh depends on the accepted maximum duration for FLISR use case. If the FLISR use case accepted maximum duration is 2 minutes, consider positioning 20-30 devices. If the FLISR use case accepted maximum duration is 5 minutes, consider positioning 40-60 devices. Above design attempts to reduce congestion as much as possible and is expected to deliver a better FLISR performance.

Scaling Up – Adding more nodes

CR Mesh should be able to handle scale much more than what was mentioned in above tables. Still, all the planning that has been done so far has been with 40% of the mesh capacity for application goodput communication. Additionally, time buffer has been considered while trying to derive the number of DA devices in the mesh. It is recommended to start with 50% of recommended scale, evaluate the mesh performance, profile the mesh utilization for the applicable use cases, and then take data driven incremental steps to add more nodes.

Note: The more the mesh is designed to reduce congestion and retransmission, better the performance would be.

Given the lossy nature of the radio medium, it is recommended to design for moderate scale and achieve better performance, than to design for high scale and suffer low performance.

Places in the Network

This section describes the Smart Grid Multi-Services Field Area Network solution's building blocks for Distribution Automation applications. The Cisco FAN solutions offers end-to-end physical and network security to detect unauthorized access to the Distribution Network's assets and to prevent access to DA communication infrastructure, as well as network services such as QoS and multi-services (DA, AMI, DER and multi-tenant). The solution offers a centralized network management system that simplifies network and device provisioning through automation process called ZTD and an advanced graphical interface for large-scale network visualization and management. In addition, customers can deploy and manage their own or third party applications at the edge in the NAN block, enabling new services and functionality to the existing Distribution grid assets.

Figure 42 Cisco FAN Solution Building Blocks

Neighborhood Area Network

The NAN is the last mile of the network communication infrastructure connecting the Utility Distribution Infrastructure assets for DA, AMI, and Remote Workforce management to the rest of the company's communication infrastructures: the Substation WAN and Data Center or Control Centers. The grid equipment, which includes feeder capacitor banks, voltage regulators, reclosers, end-of-line meters, and the transformer meters area, is connected to Cisco FAN Solution DA gateways devices such as IR510, IR800, and IR1100 using serial interfaces RS232 or RS485 or Ethernet for newer grid equipment. Cisco offers two solutions based on standard technologies:

■Cisco Resilient Mesh, which leverages the unlicensed ISM spectrum in the 900 MHz band, IEEE 802.15.4g/e, and 6LoWPAN for customers interested in deploying a private solution that offers low OpEx.

■Public Cellular DA, which uses standard public cellular 3G/4G technology and requires a monthly cellular service in order to function.

Data from the electric grid devices is transported using one of the Cisco FAN solution to different aggregation, which represent exit points out of the NAN block. The aggregation points are referred to as Field Area Routers (FAR), which use the Cisco modular CGR 1000 series router to forward the traffic upstream towards the utility energy management systems. The FAR devices provide an interface to the WAN block that acts as the backhaul for NAN.

This infrastructure, which enables customers to monitor and control the Distribution Network and perform measurement of electricity consumed and produced by prosumers, provides the foundation for advanced applications like Distribution Automation, Distributed Energy Resources (DER), and Demand Response (DR), which is a program for optimizing energy usage during peak periods.

NANs also serve as a foundation for future virtual power plants, which are comprised of distributed power generation stations, residential energy storage (for example, in combination with electric vehicle charging), and small-scale trading communities.

In the AMI scenario, the connected grid endpoints in the NAN are the smart meters that are part of the mesh radio network. These smart meters are IP-enabled grid devices with an embedded IPv6-based communication stack that are powered by the Cisco IPv6 CG-Mesh SDK.

Refer to the Cisco Developer Network (CDN) to learn more about IP-enablement for partner technologies.

Wide Area Network

The WAN tier is responsible for providing the communications overlay between the NAN block through their FARs and Data Center or Cloud Services block. The Cisco FAN solution is agnostic to the customer's WAN infrastructure. This may be:

■Private high speed in the case of the Substation On-Net, which leverages customers' dark fiber infrastructure

■Off-Net, which is based on public services offered by Service Providers like Layer 2 Carrier Ethernet or Layer 3 MPLS service combined with a VPN design to ensure privacy and security

Popular WAN backhaul options are Fiber, Ethernet, and Cellular 3G/4G. Other backhaul types for FARs may be a public or private IEEE 802.11 Wi-Fi smart cities infrastructure, satellite, or Private Cellular (if utilities have purchased cellular spectrum).

The architecture assumes established network connectivity from the NAN to the headend systems. Data security forwarded over the WAN is critically important; incorporating IPSec tunnels in the design helps meet this requirement.

Data Centers

The Data Center block represents all customer Control Center or Energy Operational Center locations that are used to manage the Distribution Network. At a minimum, a customer will have a Primary Control Center and a Back-up Control Center for disaster recovery scenarios. For large utilities, the Data Center block could have a hierarchical layout with an additional layer besides the Primary/Back-up Control Centers. This secondary layer contains the regional Control Centers that only manage a subset of the entire Distribution Network. The EOC hosts all applications necessary to operate, manage, and secure the FAN sites and equipment. It is typically located in the Utility facility and may be co-located with the Utility IT Data Center in some cases. In a multi-service FAN deployment, the EOC will host the applications and associated servers performing the tasks required by DA, AMI, DER, Remote Asset Management, and Remote Workforce Management.

Cloud Network

With the recent trends in the IT industry, customers can leverage new services that are based on Cloud Services offered by different companies. Small utilities that don't desire to maintain a Data Center infrastructure and prefer to subscribe to Cloud Services could implement Cisco Cloud Services like Jasper or Kinetic for the FAN solution.

Application Flows

The communication infrastructure is designed to support the different Energy Systems application flows. In general, the customer energy systems are located in the Data Center or Control Center locations and use a Publish/Subscriber application architecture to monitor and control the Distribution Grid assets in the NAN. The data exchange flows between a Primary station located in the DC and Subordinate stations located in the NAN over the WAN infrastructure. FlexVPN technology makes the solution agnostic to the WAN transport infrastructure and ensures that data is secure at it travels the WAN.

Figure 43 depicts the following main application flow types:

1. Publish-Subscriber between DMS and grid assets

2. Publish-Subscriber between DMS and Edge Compute Applications and between Edge Compute Applications and grid assets

3. Publish-Subscriber between DMS and Substation RTU and Substation RTU and grid assets

4. Publish-Subscriber between grid assets using peer-to-peer communication

Figure 43 Application Traffic Flow Types

FAN Layer Infrastructure Components

FAN IEEE 802.15.4g/e Devices

The FAN DA solution uses the following products to build a radio mesh network based on the IEEE 802.15.4g/e standard. The radio network requires a radio mesh Personal Area Network (PAN) coordinator. This function is supported by the solution FARs. Cisco offers CGR1240 or CGR1120 that customer can use as FAR devices.

One big difference between the two products is that the CGR1240 is an IP67-rated device that can be deployed outdoors, whereas the CGR 1120 is rated as an IP30 device and is targeted to indoor substation installations. The other main difference is the number of modular slots. The CGR1240 has four module slots that give customers the ability to deploy multiple services or backhaul interfaces; the CGR1120 only has two slots.

Lastly, the CGR 1240 obtains its power by AC while the CGR1120 supports both AC and DC.

Note: This solution design recommends the software 15.7.03 and mesh 6.0.19.

Note: For additional information on CGR1240, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/routers/1240-connected-grid-router/model.html

Note: For additional information on CGR1120, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/routers/1120-connected-grid-router/model.html

The new Connected Grid Module (CGM) Orthogonal frequency-division multiplexing (OFDM) module supports OFDM modulation and data rates based on the IEEE 802.15.4g Option 2, which is standard for DA FLISR and Volt/VAR applications.

Figure 44 CGM WPAN OFDM Module

Besides the CGM module, customers can add the Cisco Content Switching Module (CSM) to enable Edge Compute on the aggregation points of the mesh network.

Figure 45 CGM Edge Compute Server Module

Note: For additional information on CGR CSM Modules, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/iox/212038-Configure-CGM-SRV-IOx-Module-on-CGR1xxx.html

FAR routers can be also configured with 4G modems for backup network services. In case the primary backhaul link fails, the traffic from the field devices can be transported back to the Control Center or over a Public Cellular network using an overlay VPN (FlexVPN) to secure the data over the service provider network.

Figure 46 North America CGR 4G Cellular Modules

Note: For additional information on CGR CGM Cellular Modules, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/routers/1000-series-connected-grid-routers/datasheet-c78-730624.html

The IR510 is the next-generation DA Gateway that connects utility grid assets to the Cisco Resilient Mesh network. It supports higher data rates based on OFDM modulation and is best suited for DA applications. It has dedicated hardware resources for Edge Compute applications, so that customers can deploy their own applications at the edge of the network.

Figure 47 IR510 DA Gateway

Note: For additional information on IR510, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/routers/500-series-wpan-industrial-routers/datasheet-c78-730550.html

When grid assets are not close to each other enough, for example rural area with long feeders, or no clear line of sight exists between nodes, customers can deploy the Cisco Range Extender to provide additional signal coverage. The latest range extender product release is the IR530, which supports the higher OFDM data rates.

Figure 48 IR530 Range Extender

Note: For additional information on IR530, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/routers/500-series-wpan-industrial-routers/datasheet-c78-740201.html

FAN IEEE 802.11 (Wi-Fi) Devices

Cisco IoT Wi-Fi products can be used for DA use cases that require very low latency and high bandwidth; for example, FLISR with IEC61850 Goose messaging. IoT Wi-Fi products can be used to extend backhaul connectivity for Resilient Mesh FAR devices as highlighted in Figure 54.

Notes:

For additional information on IW3702, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/wireless/industrial-wireless-3700-series/datasheet-c78-734968.html

For additional information on AP1572, refer to the following URL:

■ https://www.cisco.com/c/dam/en/us/products/collateral/wireless/aironet-1570-series/datasheet-c78-732348.pdf

FAN Cellular Devices

For utilities companies that are interested in also deploying Cellular Services to Distribution Automation applications, Cisco offer a variety of products that address specific customer requirements. For example, some customers might have some power budget limitations on overhead line pole location and require DA Gateways with low power consumption.

Notes:

For additional information on IR1101, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/routers/1101-industrial-integrated-services-router/model.html

For additional information on IR807, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/routers/807-industrial-integrated-services-routers/model.html

For additional information on IR809, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/routers/809-industrial-router/model.html

For additional information on IR829, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/routers/829-industrial-router/model.html

For comparison information on Edge Compute products, refer to the following URL:

■ https://developer.cisco.com/docs/iox/ - !platform-support-matrix/platform-support-matrix

WAN Layer Infrastructure Components

The FAN DA solution uses Cisco FlexVPN as an overlay technology to connect the FAN layer to the Control Center layer leveraging an existing substation WAN infrastructure or Service Provider Cellular services. Depending on the DA design type, the VPN tunnel is established directly between the FAR router or the Cellular DA Gateway router and the Control Center Headend Router (HER).

For cases where the DA FAN traffic needs to be routed and processed within the closest substation, the design would require an additional routing device (ISR4400 or ASR1000) to perform the MAP-T address translation and act as the FlexVPN spoke devices instead of the FAR device.

Substation Network Services

When address translation is required (MAP-T domain per substations), customers can choose between the Cisco ISR4400 series and the Cisco ASR1000 series. Customers can also use virtual routers (CSR1000v) as well if compute resources are available within the substation.

Note: If customers already have routers that can provide these services, then no additional equipment within the substation needs to be acquired.

Notes:

For additional information on Cisco ISR4331 router, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/routers/4000-series-integrated-services-routers-isr/data_sheet-c78-732542.html - Product-Specifications

For additional information on Cisco ASR1001-X router, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731632.html?cachemode=refresh

For additional information on Cisco CSR1000v router, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/routers/cloud-services-router-1000v-series/datasheet-c78-733443.html

The following components are optional and recommended as best practices to increase the security of the substation network and provide additional services at the edge of the network.

Customers can deploy security services within the substation network to build a layer of protection between the utility substation WAN, LAN, and FAN layers. A variety of hardened firewalls like Cisco ISA 3000 or ASA5506H-X and non-hardened ASA5500 firewall appliances for substations with HVAC systems are available.

Notes:

For additional information on Cisco harden ISA3000 firewall, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/security/industrial-security-appliance-3000/model.html

For additional information on Cisco harden ASA5506H-X firewall, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/security/asa-firepower-services/ruggedized.html

For additional information on Cisco non-harden Next-Generation firewalls, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/security/firewalls/index.html - ~products

Besides Security services, customers can also deploy substation hardened Compute Gateways using the IC3000 appliance. IC3000 is a dedicated device for Edge Compute applications within the substation layer. Additional appliances can be deployed to meet the growing demand for compute processing at the edge of the network.

Figure 49 Cisco IC3000

Note: For additional information on Cisco harden IC3000 Compute Gateway, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/routers/3000-series-industrial-compute-gateways/datasheet-c78-741204.html

WAN Control Center

In the Control Center, HER or routers will aggregate all the FlexVPN tunnels from the remote FAR devices or Cellular DA Gateways. More details can be found in Cisco Headend Router Overview.

Headend Layer Infrastructure Components

Application Layer

The Cisco Resilient Network Headend Infrastructure is flexible and modular where it can be integrated in any Utility Control Center network infrastructure. The Headend infrastructure can be deployed as a new infrastructure block separate from the existing utility grid control systems and only allow the telemetry communication and access to the Cisco Resilient network management system to flow between the two systems. Further, some of the headend components can be re-used if the utility has already deployed them. For example, Certificate Infrastructure, IP Address, and Name management tools.

Figure 50 Headend Infrastructure Components

Note: The Headend Infrastructure design is outside of the scope of this document. Please check the SalesConnect for Cisco Field Area Network Full Headend Implementation Guide. If you do not have access, please reach out to your local sales account team.

■ https://salesconnect.cisco.com/open.html?c=db570d3f-3212-4659-a306-5f65aeab862b

Certificate Infrastructure Overview

The Cisco solution requires a Public Key Infrastructure (PKI) infrastructure to deploy and manage the solution's devices certificates. This approach provides a secure communication infrastructure between components that can also scale to very large number of devices. The Cisco solution requires two types of certificates: RSA for devices with lots of memory: HER or CGR routers and ECC for constrained devices with limited hardware resources: IR510/530, AMI meters. Also, ECC certificates are smaller in size and require less effective bandwidth utilization in the mesh network. Cisco supports the Microsoft Certificate Authority Services part of the Microsoft Windows Server 2016 and 2012.

User and Device Security Management Overview

In order to manage the system users and devices a database service is required. The Microsoft Active Directory services is one of the supported options. It can be easily integrated with Certificate Authority Services.

In addition, a Network Policy Service (NPS) is required to perform device authorization. The Cisco solution uses Cisco Identity Services Engine (ISE) since it offers additional capability over other products, such as ease of integration with the rest of the Cisco infrastructure and VPN services. Alternatively, utilities can use Microsoft NPS services.

Note: For additional information, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/security/identity-services-engine/index.html

Address and Name Management Overview

In order to ease deployment and address large-scale requirements, Cisco DA solution requires a Dynamic Host Configuration Protocol (DHCP) service to dynamically allocate IPv4 and IPv6 to the infrastructure. A DNS service is also required to easily identify the solution components. DNS plays an important role for Edge Compute applications. Cisco Network Register (CNR) offers both services and has the preferred components. Other services can be used as long as they meet the solution requirements.

Note: For additional information, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/cloud-systems-management/network-registrar/index.html

Field Network Director Overview

The Cisco Field Network Director (FND) is the main solution management system that provides Fault, Configuration, Accounting, Performance, and Security (FCAPS) services.

Note: For additional information, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/cloud-systems-management/iot-field-network-director/index.html

Hardware Security Module or Software Security Module Overview

The FAN solution requires a security module to digitally sign the IPv6 CSMP messages between the FND and Field Devices (IR510) to provide message authenticity. Customers can use a hardware appliance like SafeNet for the best level of security or leverage software functionality as a low-cost alternative. The SSM is licensed under the FND product.

Note: For additional information on SafeNet HSM, refer to the following URL:

■ https://safenet.gemalto.com/data-encryption/hardware-security-modules-hsms/safenet-network-hsm/

Tunnel Provisioning Server Overview

In order to ease the deployment of the solutions and automate the process, Cisco has developed the Tunnel Provisioning Server (TPS) to help provision the initial VPN configuration for the FAR devices. Since it communicates with devices in less secure zones, it acts as a proxy configuration services between the FAR devices and the Cisco FND.

Registration Authority Overview

Cisco routers provide Registration Authorization (RA) services for the initial device onboarding with the Certificate Infrastructure. Network components such as the FAR, Field Devices, and the NMSs receive the proper certificates once they have been authenticated and authorized.

Note: For additional information, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/routers/4000-series-integrated-services-routers-isr/index.html

Compute Infrastructure Layer

The Headend software components can be installed on any type of compute infrastructure as long as it follows the product installation hardware requirements. For customers that use Cisco UCS servers or are new to Cisco's server products family, the next section will provide guidance on product selection for installing the various FND components.

Cisco UCS Servers Overview

Note: The data in the table below represents reference numbers for general guidance. For more accurate information, please contact the Cisco account team for the latest product capabilities numbers.

For additional information on FND 4.3 Components Hardware requirements, refer to the following URL:

■ https://www.cisco.com/c/en/us/td/docs/routers/connectedgrid/iot_fnd/release_notes/4_3/rn-iot-fnd-4-3.html - 88464

Notes:

For ordering and server configuration information on UCS C220 M5, refer to the following URL:

■ https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/c220m5-sff-specsheet.pdf

For ordering and server configuration information on UCS C240 M5, refer to the following URL:

■ https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/c240m5-sff-specsheet.pdf

For ordering and server configuration information UCS C480 M5, refer to the following URL:

■ https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/c480-m5-high-performance-specsheet.pdf

Network Layer

Customers can leverage the existing switching and routing infrastructure within their Control Center or acquire new equipment to build a dedicated environment for the FAN DA solution. For the latter case, customers can select products listed in the next sections.

Switching and Routing Infrastructure Overview

Cisco has a large portfolio of switching products for Enterprise as well as Industrial customers. For Control Center switching infrastructure, the Catalyst 9200 switching family is a great platform to use because of its cost and the support for network automation (SD-Access). The Catalyst 9200 switches offers two types of chassis: one that is modular and will allow customers to upgrade the uplink ports down the road and another one with a fixed configuration. Both versions are stackable, which allows the platform to be used for small or large Control Center networks.

For Control Centers interconnected by a private WAN and where customers would like to distribute precision timing from the Control Center to remote Substations, the Cisco IE5000 switches can be used to complement the Catalyst 9200 switching infrastructure. The Cisco IE 5000 switch can be connected directly to a GPS interface and, through an additional license, can work as a PTP Grandmaster.

Notes:

For more information on Catalyst 9200 switches, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9200-series-switches/nb-06-cat9200-ser-data-sheet-cte-en.html

For more information on IE 5000 switches, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/switches/industrial-ethernet-5000-series-switches/datasheet-c78-734967.html

Cisco Headend Router Overview

To secure communication over any type of utility WAN implementations, the Cisco DA solution uses an overlay VPN technology. Therefore, the solution becomes agnostic to the transport infrastructure, maintaining the same design functionality, which simplifies the manageability of the network.

The HERs terminate the VPN tunnels from the FAN devices, in particular the FAR. For large deployments, multiple HERs can be configured to operate in a cluster. Cisco has a large portfolio of WAN routers that match any deployment size of the DA solution.

Customer can choose different models from the same routing product family for consistency and ease of management. The following products can be use as HER devices.

Note: The data in the table below represents reference numbers (single dimension) for general guidance. For more accurate information, please contact the Cisco account team for the latest product capabilities numbers.

Note: For CSR1000v product details, please reach out to your Cisco sales account team.

Network Security Devices Overview

Network Firewalls and end nodes agents provide good security boundaries between the Grid Operation Systems as well as the different components of the Cisco HE that communicate with devices outside the utility physical security boundary. Cisco offers a comprehensive portfolio of security network appliances and software applications.

Note: The data in the table below represents reference numbers (single dimension) for general guidance. For more accurate information, please contact the Cisco account team for the latest product capabilities numbers.

Notes:

For additional information on ASAv, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/security/adaptive-security-virtual-appliance-asav/datasheet-c78-733399.html?cachemode=refresh

For additional information on NGFWv, refer to the following URL:

■ https://www.cisco.com/c/en/us/support/security/firepower-ngfw-virtual/tsd-products-support-series-home.html

For additional information on FRP2110, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/security/firepower-ngfw/datasheet-c78-736661.html

For additional information on FRP4110, refer to the following URL:

■ https://www.cisco.com/c/en/us/products/collateral/security/firepower-ngfw/data_sheet-c78-736661.html

Utility SCADA Systems Architecture Overview

Traditionally, SCADA system architecture is centralized and uses a single tier between the SCADA Primary stations that are located in the Primary and Back-up Control Centers and the SCADA clients (which are typically RTUs located in the substations). This architecture worked well as communication technology evolved and made its way onto the distribution grid network for asset monitoring and control. Figure 51 is an example of such system.

Figure 51 Traditional, Centralized SCADA System Architecture

However, with the introduction of Distributed Energy Resources (DER) and new requirements to increase the reliability of the distribution grid, customers in both the European and North America markets have started considering a two-tier SCADA architecture, which is sometimes called the Distributed SCADA architecture. It moves some of the decision logic from the Control Centers closer to the substations in order to achieve better system reaction times and make the system more flexible to changes in the grid.

Figure 52 Distributed SCADA System Architecture

In the next couple of sections, you will learn about the different Cisco FAN DA designs available for implementation that will support both types of SCADA architectures. The Private Network solution is a better option if you have a distributed SCADA architecture or believe you will be moving towards one in the near future.

Cisco DA Feeder Automation Solution based on Standard Unlicensed 900MHz ISM Band

Utility customers that prefer a private solution can opt to run their Distribution Automation systems using only the Cisco Resilient Mesh design based on the ISM open license 900 MHz band. This design eliminates any dependencies on third party services and allows customers to operate the network from provisioning to configuration management to the troubleshooting point of view. It offers a lower operation cost model since there is no monthly reoccurring service cost for the transport service and it eases the regulatory compliance process. It is also less vulnerable to weather-related or crowding events that affect a cellular service. This design is well suited for large or small utilities and it can be deployed in urban or rural territories. It supports both centralized and distributed SCADA systems.

For utilities with a distribution substation fiber-rich WAN network, the FAR can be installed within the substation yard. This type of implementation is better suited for Distributed SCADA systems or future Smart Grid implementations where the Grid control needs to be regionalized for fast control response.

The design can leverage a VPN service over the existing substation WAN if encryption is required or if the WAN is not IPv6 capable.

Figure 53 DA Design Based on 900MHz ISM Band Solution [4003] 256299 or not?

For substations that don't have a high-speed WAN connection, customers can choose to use the Cisco Outdoor Wi-Fi mesh solution to backhaul the Resilient Mesh traffic towards the closest substation that has a fiber WAN connection.

The Wi-Fi backhaul can use a mesh topology similar to the Resilient Mesh or Point-to-Point Wi-Fi links to extend the substation connectivity. An additional benefit of this design is that customer can deploy DA applications that require high bandwidth and low latency like ultra FLISR based on IEC61850 or microPMUs. The FARs can be installed anywhere in the field where the utility has assets.

Figure 54 Wi-Fi Backhaul for Location without Fiber

Cisco DA Feeder Automation Solution using Public Cellular Service (3G/4G)

As an alternative technology to the 900 MHz ISM band, customers can use a design that is a based only on the Cisco Cellular products for DA applications. This design has a higher OpEx due to reoccurring Cellular services, but for certain countries where the 900MHz band is not supported, Public Cellular is the only reasonable option.

Besides cost, customers need to take into consideration the cellular base station congestion, which can be temporarily caused by crowded events or permanently caused by over-subscription when cell has peaked its original planned size. Public Cellular service does not offer service priority among its tenants. Cellular technology uses asymmetric bandwidth, which was designed to offer higher download link speed than upload link speed. The DA traffic profile is opposite to the Cellular Service link design where most of the traffic and bandwidth requirements are using the upload link speed. The SCADA primary system makes small size requests and the grid field device responds with larger amounts of data.

Each distribution automation grid asset will be connected to a Cisco DA gateway that will build a VPN tunnel over the Service Provider Cellular network towards the Control Centers to encrypt the data. Service Providers support both hub-and-spoke topology as well as an any-to-any topology for peer-to-peer communication. The Cisco VPN services can be also configured to match the Service Provider service topology for optimal traffic flow. This design will accommodate both Centralized and Distributed SCADA implementations. For the latter scenario, a cellular DA gateway might be required to be installed within the distribution substation even if the substation has fiber or high-speed connectivity. The traffic from nearby field devices could share the same cellular base station with the Substation DA Gateway, which reduces the latency communication for a two-tier SCADA implementation. The Substation DA Gateway could also be used as a back-up solution for the substation primary WAN link. Without cellular connectivity, the traffic between grid field devices to nearby substation RTUs will hairpin at the Control Center since that is the only place that interconnects the Fiber and Cellular networks.

Figure 55 Cellular DA Design

Cisco DA Feeder Automation based on Hybrid Design: Cellular & 900MHz ISM

For customers that prefer a Distribution Automation design based on private provider-managed Cellular services, but with a territory where the cellular coverage is not well developed or has pockets of no-service coverage, or where certain locations requires more bandwidth than what the ISM band can offer, a hybrid DA design with cellular and Resilient Mesh is more appropriate. The 900Mhz solution will complement the cellular by allowing customers to automate feeders that are located in new development areas or rural areas that lack cellular service. Customers can install the FARs closer to the edge of the cellular coverage area and extend the network connectivity services using Field Devices and the ISM 900 MHz band. Figure 56 is an example of such deployment.

Figure 56 Hybrid DA Design

End Devices Connectivity

Today, most of the Distribution Automation grid devices are connected to the Resilient Mesh communication equipment via legacy, low speed, and asynchronous serial interfaces RS232 or RS485. In order to transport the serial traffic from grid devices to the grid management systems located in the Control Centers over an IP network, Resilient Mesh uses a feature called Raw Socket to encapsulate the serial traffic in IPv4 packets over a UDP or TCP session. In this case, each field device (IR510) requires an IPv4 address to establish a session with the SCADA DMS servers or other terminating devices.

Figure 58 Serial Grid Device Connectivity

Distribution grid devices that support Ethernet interfaces can be connected to the IR510 Ethernet 0 port using IPv4 or IPv6 protocol.

Figure 59 Ethernet Grid Device Connectivity

With the introduction of Edge Compute capabilities, customers can now run grid applications local to the distribution grid devices. These enable customers to perform more advanced grid operations where legacy grid devices have limited capabilities. In addition, customers can leverage IoT architectures and protocols like Constrained Application Protocol (CoAP), MQ Telemetry Transport (MQTT), and Data Distribution Service (DDS) to deploy a middleware message bus where data manipulation can be performed. For example, data from different distribution grid device types can be normalized and exchanged between multiple devices or systems across the Resilient Mesh. For these scenarios, the Edge Compute application container will require additional IPv4 addresses and IPv6 in the future.

Note: Customers are responsible for the developing or acquiring third party Edge applications that can be hosted on the Cisco FAN DA devices. Cisco provides a management solution for deploying and monitoring such applications at scale.

Figure 60 Edge Compute Application Connectivity

IP Address Schema

FAN Resilient Mesh Layer

Mesh Address Space

The Cisco Resilient Mesh solution was developed with a forward-looking mindset to overcome IPv4 address exhaustion as well as provide support for millions of end devices and sensors. Therefore, routing within the mesh is based on native IP6. Since IPv4 is still prevalent for the end devices, especially in the Utility Industry where adaption of IPv6 is in its infancy stage because of lack of equipment vendor support, the Resilient Mesh also leverages industry standards for address translation between IPv4 and IPv6 as traffic flows over the mesh.

In summary, the mesh requires both IPv4 and IPv6 networks in order to transport the end grid devices traffic.

Design Guidance:

The size of the mesh network address space, number of subnets will be based on the following factors:

a. Utility number of end devices: capacitor bank controllers, recloser, etc.

b. Utility number of take out points or exit points out of the mesh. This depends on the utility's substation WAN fiber presence as well as end device density for an area since the ratio of Field Devices per FAR needs to be considered.

c. Number of Field Devices per Mesh PAN ID, including expected number of devices that would perform PAN migration

d. Number of mesh range extenders required to close any RF signal coverage gaps

e. Number of edge compute applications and number of locations where application(s) will be deployed

f. Future growth, additional Smart Grid use cases: DER, etc.

In general, each Cisco mesh device requires the following address types based on the design redundancy and resiliency requirements, substation WAN support for IPv6, and customer requirements for edge compute applications

IPv6 as IPv4 uses two type of network addresses:

■Globally Unicast Addresses (GUA)

■Unique Local Addresses (ULA)

Global IPv6 prefix: Obtained through one of the five Regional Internet Registries (RIR): AFRINIC, APNIC, ARIN, LACNIC, or RIPE. The entity requesting the prefix from the RIR must be registered with the RIR as either a Local Internet Registry (LIR) or end-user organization. As an alternative, a global prefix might be obtained from an ISP.

A utility should consider registering as a LIR to obtain its own IPv6 prefix and therefore be fully independent from any churn in the ISP addressing architecture.

RIRs define policies regarding allocation of an IPv6 prefix and the prefix size. A RIR prefix allocation is by default ::/32 prefix for a LIR, and ::/48 for an end-user organization. The RIR policies also define how larger or smaller prefixes can be allocated to a LIR and an end-user organization.

Note: For additional information on RIR, refer to the following URL:

■ https://www.ripe.net/publications/docs/ripe-707

A justification that is based on the number of sites and hosts must be given for the non-default allocation. The number of FAN sites and subnets drive the decision to register as a LIR or as an end-user organization and further justify the requests made for prefix allocation and size.

ULA IPv6 prefix: A unique local address (ULA) IPv6 prefix, documented in RFC 4193, is allowed to be "nearly unique." It starts with a FC00::/7 value but the following 41 bits, global routing ID, allow an addressing space far greater than the 3-private IPv4 prefixes (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) documented in RFC 1918. The size of the global routing ID effectively produces a pseudo "uniqueness." Note, however, that no central registration of ULA prefixes currently exists.

The main differences between selecting a global or ULA IPv6 prefix are the following:

■A global prefix requires registration to the RIR either as LIR or as an end-user organization. This requires paper work and fees before getting and justifying an IPv6 prefix allocation. A ULA does not require this registration.

■Filtering at the border of the utility routing domain:

■A ULA IPv6 prefix must NEVER be advertised to the Internet routing table.

■A global IPv6 prefix or portions of its address space might be advertised to the Internet routing table and incoming traffic MUST be properly filtered to block any undesirable traffic.

■Internet access: A ULA-based addressing architecture requires the IPv6-to-IPv6 Network Prefix Translation (IPv6 NPT, RFC 6296) device(s) to be located at the Internet border. Remote workforce management use cases, such as third-party technicians connecting to their corporate network from a FAN site or an FND operator using the Google map features, might require Internet access. For web access, web proxies can be a solution.

■Once an IPv6 prefix has been allocated for the FAN, a hierarchy numbering the regions, districts, sites, subnets, and devices must be properly structured. IPv6 addressing is classless, but the 128-bit address can be split among a routing prefix, upper 64 bits, the Interface Identifier (IID), and the lower 64 bits. A hierarchical structure eases the configuration of route summarization and filtering.

Design Guidance:

Cisco supports both address space implementations and highly recommends customers that use the Global Unicast Address space to filter the DA solution's IPv6 prefixes at the company's internet security border.

For customers that choose to use ULA address space, it's good to ensure that NAT66 is not enabled between the DA IPv6 prefixes and the company internet global prefixes in order to avoid routing traffic from the internet in the DA infrastructure.

Besides the unicast addresses, the FAN DA solution also requires IPv6 Multicast address allocation. The firmware for the FAN DA devices is done using multicast to optimize the distribution of the software over the LLN.

Mesh Layer 3 Boundaries

The Cisco Resilient Mesh is a Layer 3 IPv6-routed network. Each Field Area Router (CGR) mesh radio interface (WPAN) is a Layer 3 boundary and defines a Personal Area Network (PAN) that shares the same Layer 3 network prefix among all devices associated with the that PAN ID. Therefore, the address prefix length assigned to each CGR WPAN interfaces is directly related to the maximum number of field grid devices connected to a PAN, the number of Range Extenders (IR530) within the PAN, and the expected field devices that can migrate to the PAN from adjacent PANs.

Figure 62 CGR WPAN and IPv6 Address Assignment

For FAR High Availability (HA) designs, a direct Ethernet link is required between the two CGR routers that are used as a heartbeat link to detect CGR failure and for state replication of the WPAN state as described in FAN Infrastructure Layer. The HSRP protocol runs between routers; therefore, at a minimum, the IPv4 subnet prefix should be a /29.

Address Translation

For scenarios where the end grid devices are connected via serial interfaces or the Ethernet using IPv4 to the Field Devices (IR510), address translation is necessary to translate IPv4 packets into IPv6 packets and vice versa and allow end-to-end communication between islands of IPv4 networks over the Mesh IPv6 network, as shown in Figure 63. Tunneling is another method, but due to additional overhead associated with encapsulation, it's not suited for Low power and Lossy Networks (LLNs).

Figure 63 Address Translation (IPv4 and IPv6)

The Cisco Resilient Mesh network uses the IETF Network Address Translation and Protocol Translation (NAT-PT), in particular MAP-T, which is a double stateless NAT64 translation. The benefit of stateless address translation is that Cisco devices don't have to maintain a translation state and also allow DA devices to migrated within mesh without affecting the traffic flow over new paths.

Note: Currently, the software version 6.0 running on the IR510 support up to 15 unique NAT44 entries.

The MAP-T uses some key concepts: MAP-T domains, mapping rules: Default Mapping Rule (DMR), Basic Mapping Rules (BMR), Forwarding Mapping Rules (FMR), and Border Router (BR).

Note: For additional MAP-T information, refer to the following URL:

■ https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/15-mt/nat-15-mt-book/iadnat-mapt.pdf

Design Guidance:

Based on the utility SCADA architecture and the traffic patterns, MAP-T can be implemented in two types of designs:

–Single MAP-T domain per HER, which is suited for Centralized SCADA where all traffic from the field grid devices flow to the Control Center SCADA systems, and peer-to-peer communication between grid devices is not required or supports high latency requirements. Depending on the FlexVPN design, the HER will learn from FAR routers (CGR) each PAN host routes (/128) to the IR510 devices. If WAN FlexVPN is not configured, spoke-to-spoke communication traffic will hairpin at the Control Center. This design is suited for small-to-medium size networks with thousands of grid devices.

–One MAP-T domain per substation, which is optimal for use cases like FLISR where peer-to-peer communication is required and has low latency requirements; therefore, the traffic does not have to hairpin at the Control Center HER device. This design requires an additional routing device in the substation to perform the Border Router functions. The advantages of these design is that it scales to millions of devices, has optimal routing, and host routes are not advertised in the WAN block, only summaries.

Figure 64 Single MAP-T Domain

Figure 65 Multiple MAP-T Domains

Recommendation:

For grid equipment upgrades or new grid equipment installation, customers should look at selecting vendor equipment that supports IPv6 protocol or at least work with the grid equipment vendors to add IPv6 capabilities to their devices. This will help the industry to adopt faster IPv6 protocol and simplify the FAN DA design by allowing end devices to communicate natively using IPv6.

Wide Area Network Layer

The FAN WAN layer uses a VPN overlay design to make the solution agnostic to the different utility customer distribution WAN network designs. The VPN services runs between the FAR devices or the Substation WAN routers and the HERs located in the Control Center.

Customers can allocate any private IPv4 and IPv6 address space as long as the space is unused within the company. The best approach is to configure the VPN devices in a dual-stack configuration.

The size of the address space depends on the following factors:

■Number of FAR devices (CGRs)

■Number of Control Centers

■Redundancy requirements for the HER within a Control Center

■Ratio of FARs per HER device

Note: For additional information regarding DA WAN FlexVPN design, refer to the following URL:

■ https://www.cisco.com/c/en/us/td/docs/solutions/Verticals/Distributed-Automation/Secondary-Substation/DG/DA-SS-DG/DA-SS-DG-doc.html

Headend Layer

The Headend infrastructure address spaces depends heavily on the existing IP address schema used in the Control Center. In general, the new FAN DA Headend infrastructure requires both IPv4 and IPv6 address space.

Note: For additional information regarding the Cisco Field Area Network Full Headend Implementation Guide, refer to the following URL. If you don't have access to the resource, contact the local sales account team.

■ https://salesconnect.cisco.com/ - /search/headend/content

Fragmentation and Reassembly

FAN Resilient Mesh Layer

End Devices MTU

The grid devices connected using the Ethernet interfaces use a Maximum Transmission Unit (MTU) of 1500 Bytes. The Ethernet standard does not support fragmentation and it is the responsibility of the sender device to fragment packets over 1500 Bytes. If a link has a smaller MTU between the sender and receiver devices, in case of VPN designs or Cellular transport service, then the network device in the path will perform network fragmentation, which is an intensive resource process that causes network service degradation or packets to be simply dropped.

With IPv6 based on RFC8200, network devices do not perform fragmentation anymore; the RFC guidance is that the minimum IPv6 MTU configurable on an interface is 1280 bytes, which means any packets smaller than that are almost guaranteed not to be fragmented. For transport interfaces with MTU lower than 1280 bytes, an additional protocol must be implemented to perform fragmentation and reassembly below Layer 3. This was the case with the initial IEEE 802.15.4 where the Layer 2 MTU was 127 bytes, and the mesh was using the 6LoWPAN Adaptation Layer to perform fragmentation and reassembly. Further, upper layer protocols and applications should not send packets larger than 1500 Bytes unless they are sure the receiver can reassemble packets above 1500 Bytes.

In order to avoid these type of scenarios, customers acquiring new grid equipment should ensure at minimum that the equipment can interpret IPv4 ICMP message type 3 - "Destination Unreachable" with Code 4 - "Fragmentation Needed and Don't Fragment was Set" from DA Gateways and fragment the packets in smaller sizes based on the lowest Path MTU in the ICMP message. Ideally, utilities should buy equipment that has Path Maximum Transmission Unit Discovery (PMTUD) capabilities. PMTUD is even more critical for grid equipment supporting IPv6 communications due to the lack on network fragmentation.

For legacy grid devices that do not support PMTUD, the customer can manually set MTU on the device interface to the smallest value of the MTU on the communication path after taking into consideration all protocols headers overhead in case of the WAN VPN. For protocols like DNP3, this is not a concern since DNP3 data link MTU is 292 Bytes.

Resilient Mesh MTU

The DA Gateways and FAR devices support PMTUD over the mesh radio and can signal to end devices when the application needs to reduce the packet size to avoid network fragmentation.

Originally, the IEEE 802.15.4 Physical Service Protocol Unit (PSDU) maximum size was 127 Bytes and the Cisco Mesh was using 6LoWPAN adaptation layer that was performing fragmentation/reassembly functions between the Layer 3 IPv6 and Mesh Layer 2.

The IEEE 802.15.4-2015 standard increased the original PSDU of 127 Bytes to 2047 Bytes so the 6LoWPAN adaptation layer does not perform fragmentation and reassembly. It only does header compression for IR510 within the mesh.

Figure 66 shows that the grid devices MTU should not be set higher than 1472 Bytes to avoid packet drops if end devices do not support PMTUD. The CGR Maximum Mesh MTU is 1500 Bytes of which 28 Bytes are used by IPv6 headers. Over the WAN, assuming it supports full Ethernet 1500 MTU, the IPSec and GRE header add up to about 112 Bytes when using IPv4 as transport and 164 Bytes for IPv6 transport. To accommodate for future growth as encryption algorithms evolve, it is recommended to set both IPv4 and IPv6 MTU to 1300 Bytes.

Recommendation:

The smallest link MTU will be over the WAN FlexVPN tunnel; therefore, in order to avoid fragmentation or packet drops, customers should also set the Grid Field Devices MTU for both IPv4 and IPv6 to 1300 Bytes when devices do not support PMTUD. If devices support PMTUD, then it's best to leave the Grid field devices to use their default MTU and rely on the PMTUD features that works with UDP and TCP applications.

Note: DNP3 and Modbus use small PDU size and are not prone to fragmentation. It's possible that other device management protocols could benefits from PMTUD or manual configuration of the MTU on the Grid field device.

Note: When packets are small, it will take less airtime to transmit and be open for packet errors due to interference or collision.

Figure 66 End Device Maximum MTU for Ethernet Connectivity

Grid field devices connected via Serial interface will use the Raw Socket feature to packetize the DNP serial data over the IP network. The Raw Socket session can be configured to use UDP or TCP for transport, but UDP is the recommend configuration since Mesh has a built-in reliability function at Layer 2 as well as the DNP protocol.

A Raw Socket session from the DA Gateways can be terminated in the Control Center in multiple scenarios. For IP SCADA systems, the Raw Socket can be terminated directly on the DMS Front End Processor (FEP) system as depicted in Figure 67. When configuring Raw Socket over UDP, customers should set the Maximum Packet Length for the session to something that matches the protocol used. For example, for DNP set the Packet Length to be around 292 Bytes.

Note: To avoid DMS packet reassembly, enable the message interleaving delay on the Grid field devices serial port.

Figure 67 Raw Socket Packet Length for Serial Connectivity

Note: The default Packet Timer is 10 milliseconds.

For traffic towards the Grid field devices (downwards), the MTU packet size depends on the Mesh routing mode configuration. For Storing Mode, the same Upwards principal apply as described in Figure 68.

When the Mesh is configured for Non-Storing Mode, then the FAR devices will insert a Source Routing Header and its size depends on the depth of the mesh, the hop count to destination.

Figure 68 Non-Storing Mode MTU Considerations

WAN MTU

The substation WAN device's MTU depends on the WAN device connectivity interface type (Ethernet or Cellular) and the type of WAN transport service used (Private WAN, Managed MPLS service or Public Cellular Service).

Regardless of the type of SCADA architecture used, the WAN device physical interface MTU should reflect the WAN transport service MTU.

The FlexVPN spokes, either the FAR or substation WAN devices, and the Tunnel interface MTU should be set based on the lowest MTU of the Physical interface or Transport Service minus the FlexVPN protocols overhead. The same configuration should be applied on the HER Virtual Tunnel interface. As an example, Figure 69 shows the Tunnel MTU size when the WAN supports the full Ethernet MTU of 1500 Bytes.