Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
SOFTWARE-DEFINED WIDE-AREA NETWORKS. FOR DISTRIBUTED MICROGRID POWER SYSTEMS. BY. XUANYAO ZHANG. THESIS. Submitted in partial fulfillment of the ...
Typology: Slides
1 / 35
This page cannot be seen from the preview
Don't miss anything!
Submitted in partial fulfillment of the requirements
for the degree of Master of Science in Electrical and Computer Engineering
in the Graduate College of the
University of Illinois at Urbana-Champaign, 2018
Urbana, Illinois
Adviser:
Research Assistant Professor Sibin Mohan
ii
Cyber-physical systems have increasingly taken advantage of packet-switching networks
for control and data acquisition. A major example is the realization of the smart grid. On the
networking side, software-defined networking (SDN) has been trending for the past decade. With
the help of SDN, we are moving towards power grids that have both intelligence and security. In
this thesis, we focus on providing a versatile SDN infrastructure for power-system applications in
the environment of microgrids. We conduct simulations and collect statistics to demonstrate that
the SDN approach facilitates communications and enhances security for certain microgrid
applications.
In the electric power industry, the trend is toward smartness for better control and smallness
for better efficiency and resiliency. This trend has led to the concept of microgrid [1]. A microgrid
is a localized group of electricity sources and loads within clearly defined electrical boundaries. A
microgrid as an entity can be connected to a main grid and other microgrids in grid-connected
mode, trading energy deficit or surplus and balancing the load on a larger scale. Or it can be
isolated from others and operate entirely on its own energy resources, thus preventing the spread
of grid failure due to natural disasters or human attacks [2]. The communication side largely
follows the topology of a microgrid design, i.e., intra- and inter-microgrid communications.
A very simple microgrid may consist of a microgrid controller, a human-machine interface,
a battery system and some protective relays. SCADA (supervisory control and data acquisition)
protocols have evolved in the past four decades from serial protocols such as Modbus [3] to modern
standards that use switched networks, including Ethernet and TCP or UDP over IP. Examples of
these include IEC 61850 [4] and DNP3 [5]. SCADA operation consists of a master that polls
devices for measurements, performs calculations and possibly issues commands to devices capable
of undertaking control actions. The actions, for example, can be tripping a relay to isolate a fault,
dispatching distributed energy resources (DER) to balance the load with the main grid and other
microgrids or changing DER power input settings in response to conditions.
As the electric power sector increasingly adopts smart-grid technology, network-enabled
devices (for measurements) and control have become widespread. Reliance on common lower-
layer standards such as Ethernet and IP enables greater versatility in configuration, topology and
coordination for devices. However, it also poses challenges in connectivity and security as newer
communication protocols emerge (such as OpenFMB [6]) and the attack surface increases as
evidenced in the recent cyberattack on Ukraine’s power grid [7]. Such trends in power systems
prompt this research to leverage a new networking paradigm, software-defined networking (SDN)
[8], to better serve the infrastructural need of power system networks.
Conventional networks work in a distributed fashion where routers use well-defined
protocols to exchange messages with peers to converge to a steady state so that packets are
In section 2.1, we explain the advantages of a microgrid design and a sample topology. In
section 2.2, we discuss the need for an open standard for future devices and discuss how OpenFMB
charts a course forward. In section 2.3, we provide a high-level view of the reliable middleware
that bridges OpenFMB and the underlying network. The two sections provide us some background
knowledge of smart-grid designs and the motivation for this research. Then in section 2.4, we
introduce the concept of software-defined networking and discuss why it helps achieve the goal of
this research. Lastly in section 2.5, we prepare readers with the knowledge of the tools and
protocols in use.
Motor ZIP
Critical
Load
Substation 1
MG1 MG
POI- 1
POI- 2
Substation 2
NG PV
ESS
ZIP
Critical
Load
Diesel
ESS PV
Motor
Figure 1 A use case with two microgrids. The red and blue cubes represent reclosers, with red ones being closed
and green ones being open by default. Circular devices represent power sources and rectangular ones power sinks.
Lines and bars represent electrical interconnections. Abbreviations are as follows: MG: Microgrid; POI: Point of
Interconnection; NG: Natural Gas; PV: Photovoltaic Panel; ESS: Energy Storage System; ZIP: Constant Impedance
(Z), Current (I) and Power (P).
Diagram courtesy of Alfonso Valdes.
A traditional power grid design has a strictly hierarchical topology featuring centralized
power plants, long-distance transmission lines and substations owned by utility companies. Central
plants can take advantage of economies of scale, but this advantage has been offset by distributed
energy generation with the advent of smaller power sources (microsources). A microgrid design
takes advantage of the proximity to residential, commercial and industrial areas to achieve greater
efficiency and flexibility. Such design, with advanced controls, could possibly reduce the loss in
distant power transmission, reduce disturbances in terms of power quality, provide resiliency in
the event of outages [9] and increase efficiency through the use of waste heat [1].
An example use case is demonstrated in Figure 1. It has two microgrids, MG1 and MG2,
that are electrically connected to each other and, through substations, to the main grid. MG1 and
MG2 maintain electrical connection to the main grid but electrical isolation between themselves
by default. They can dynamically reconfigure electrical connections by using the reclosers in
response to certain conditions.
Unless one chooses to purchase power devices from a single vendor, there is a high
likelihood that a designer will encounter heterogeneous devices and different protocols.
Traditional protocols are usually single-purpose that use a centralized control where there is clear
dichotomy of masters and slaves. A conventional setup is to connect the field devices to the utility
central office one by one (point to point). The centralized entity is responsible for collecting
telemetric data from field devices for processing and issuing commands back to them. This,
however, does not provide interoperability because field devices cannot directly interact among
themselves.
The Open Field Message Bus (OpenFMB) [6] seeks to address these issues by putting forth
an open-source common information model that every field device shall conform to, either as is
for newer devices right off the shelf or through adapters for existing ones. This new model enables
distributed intelligence and scalability for deployment. Additionally, due to its distributed nature,
more robustness in connectivity and more autonomy in decision making can be achieved.
OpenFMB’s versatility comes not only from its data model (the upper layer) but also from
the variety of middleware it can be adapted to, such as the Message Queue Telemetry Transport
(MQTT), the Advanced Message Queue Protocol (AMQP) and the Data Distributed Service (DDS)
Software-defined networking (SDN) [8] has been a trending topic since its inception a
decade ago. In this section we introduce what it is and how it facilitates communication and
provides security.
Traditionally, core network switches use distributed protocols such as OSPF and BGP [13]
to learn the global topology and peer relationships. They then calculate the optimal paths to
destinations from the learned topology or relationships in order to determine which neighbors
should be forwarded the packets, subject to business relationships and regulations. Due to the
nature of these distributed algorithms, it is a slow process for the switches to converge to a
consistent state where packets are correctly and efficiently forwarded. Furthermore, any change of
state in the topology (such as a switch breaking down or a link becoming congested) will disrupt
the equilibrium and cause the (slow) convergence process to start again.
SDN seeks to eliminate the need for a distributed process by introducing a centralized
control plane as a replacement for the routing logic residing at individual switches. The controller
gathers routing messages from border gateway routers, computes the optimal solutions and then
formulates them as simple match-action rules to install onto the switches (forwarding plane). A
full-scope implementation also has an application layer that interfaces with the controller to allow
the user to change switching behaviors and monitor the state of the network. SDN speeds up
routing, eases management and provides possibility for traffic engineering that cannot be achieved
in a traditional network.
This thesis focuses on a specific application of SDN, viz., the software-defined wide-area
network (SD-WAN) because our efforts aim primarily to provide connectivity for field devices
within microgrids that constitute local-area networks (LAN) and across multiple microgrids via a
wide-area network (WAN). We also provide support for isolating individual devices or entire
microgrids to protect the power system from compromised nodes.
OpenFlow [14] has emerged as the de-facto standard protocol for SDN development. It
allows the controller to direct actions for packets based on their headers and provides a few
methods to query for the switches’ statistics, such as the number of bytes or packets that have been
transmitted or received for a given interface and the state of interfaces.
Open vSwitch [15] (OVS) provides the functionality of the SDN forwarding plane
(switches and links). It consists of several user-space utilities to configure topology, provide
OpenFlow interface for the SDN controller and interact with its kernel Datapath module that
provides low-latency, high-throughput packet forwarding.
Ryu [16] is an open-source Python framework for developing SDN controllers. It supports
OpenFlow protocol versions 1.0 through 1.5, the latest release. It also incorporates a Web Server
Gateway Interface (WSGI) component for developers to implement the application layer of the
SDN, enabling on-the-fly reconfiguration of the control plane.
Mininet [17] is an emulator that allows rapid prototyping of a large network. It leverages
the Linux kernel to create virtual Ethernet interfaces (veth) in different network namespaces and
Open vSwitch to create virtual switches that the interfaces can be attached to. With veth comes the
benefit of capturing and analyzing traffic with tools such as Tcpdump and Wireshark [18]. Mininet
also provides shell interfaces (bash) for virtual hosts through process-based virtualization so that
traffic can be generated and received in the simulation, thus providing an end-to-end testbed for
SDN implementations. Most of the testing in this thesis was conducted in the Mininet framework.
2.5.1 IP Multicast
In the world of power systems [19], multicast plays a critical role for many publish-
subscribe protocols. Many field devices do not have a great amount of computing power. The
industrial Ethernet switches do not have large throughput either unlike commercial switches at
data centers. Multicast can reduce unnecessary traffic by pushing traffic duplication toward the
last links as far as possible, which, in turn, reduces the load on the senders and backbone switches.
IP Multicast is divided into two parts, the WAN segment and the LAN segment. The WAN
side consists of routers that usually run Protocol Independent Multicast (PIM) protocols for
forwarding traffic. PIM has two modes: the dense mode (DM) [20] and sparse mode (SM) [21].
The DM postulates a scenario where the majority of end hosts in the network are interested in
certain multicast traffic, so the idea is to (a) flood the entire autonomous system with such traffic
and (b) let downstream routers notify upstream ones that they do not want such traffic when there
Most of the SDN research in the area of power systems seems to have focused on single-
grid use cases. Pfeiffenberger et al. [24] presented a way for efficient data delivery of multicast
traffic with fault tolerance and with a focus on IEC 61850 traffic. Cahn et al. [25] tried to adapt
SDN for the power grid as it transitions into smart grid, in order to provide better management,
auto-configuration and security.
Other research [26], [27] primarily deals with several different field devices within the
same power grid and seeks to identify and optimize real-time flows in such a system or to improve
security through filtering. I would argue such efforts still fall within the scope of the generic
application of SDN. To the best of our knowledge, ours seems to be the first to use SDN for
bridging traffic, both within and across microgrids.
There is a component to RTI’s DDS framework called Routing Service [28]. It is an out-
of-the-box solution to bridge DDS applications across different publish-subscribe domains and
LANs. The idea is to have a service application running in every LAN that subscribes to selected
DDS messages. The services are preconfigured with their peers’ IP addresses and listening ports,
so that messages can traverse from one LAN to another through the services that act like proxy
servers. This solves the connectivity issue but is very inefficient as messages do not go from source
to destination directly. Furthermore, this solution is limited to DDS or perhaps only RTI’s
implementation of it, thus lacking interoperability.
For commercial use, SD-WAN [29] is gradually replacing traditional services such as T-
carrier and MPLS [30]. It is mostly used to optimize traffic for multi-homed offices and provide
dedicated and secured connectivity between branch offices. Existing solutions include Cisco’s
iWAN [31] and Riverbed’s SteelConnect [32]. While they likely already provide multicast support
across branch offices, they might not allow customizability for power-system protocols on a per-
flow basis.
In this chapter, we begin by presenting a power-grid use case that reflects a real-world
scenario. Then we delve into the technical details of IP Multicast and we explain the core design
of the SDN controller that is the control logic and the forwarding rules derived from it. Finally, we
present an interface for external inputs to change the controller’s behaviors and retrieve the state
of the network, thus demonstrating a full-stack solution in SDN.
Figure 3 Topology for our experiments. “Gateways” and “routers” signify SDN controllers. Blue loops with arrows
through them signify SDN switches. Command-line interfaces in black backgrounds signify end hosts. Links are in
blue or red, respectively, representing the transport network and management network.
We simulate a physical setup that includes three microgrids inter-connected through a
WAN and two field devices residing at each microgrid. We “own” every microgrid, e.g., the
networking infrastructure and the field devices, but we do not have any control over the WAN that
is operated by third parties such as AT&T and Verizon. Figure 3 shows our simulated network in
Mininet. Switches s1, s2 and s3 act as gateways, each residing at a microgrid, numbered from 1 to
directly connected to the gateways. This improves control as we can police the traffic of any given
host with the controller for the gateways. Switches s4, s5 and s6 simulate the WAN. They have a
separate controller because an energy company does not usually own the networking infrastructure
inactive backup lest active links fail. This works fine in a small scale, physical LAN environment.
In our case, however, each northbound link is potentially across a distant geography, with limited
bandwidth and high latency. In order to preserve the optimal routes that are the direct links from
one to another, we divide interfaces into two categories, the northbound ones that are VXLAN and
the southbound ones that are end hosts (field devices). For broadcast traffic received from the
southbound interfaces, a switch floods it to all other interfaces. For broadcast traffic received from
the northbound interface, it only floods it to southbound interfaces, knowing they must have
already been flooded to other switches by the first hop gateway.
4.3.1 Event Loops
Figure 5 The control plane logic consists of four event loops. The rightmost loop is the query thread. The second
from the right is the timer thread. The two loops on the left handle IGMP join and leave messages.
The control plane logic has multiple threads to deal with different aspects, as summarized
in Figure 5. The query thread, every 60 seconds, sends out membership requests to southbound
hosts, triggering passive membership reports within the response interval (10 seconds). The timer
thread, every unit of time, increments the time counter for each listening port. If the time counter
reaches 75 seconds (the amount of time should be greater than the query interval plus the response
interval), it times out and is evicted from the control plane. A more frequent query results in a
Control Plane Logic
Unit of time lapsed
Every 60 seconds
lapsed
Upon receiving a "join" request
Event-driven Loops
Reset timer
Host already
exists?
Evict port from
control plane
Repeat
Add port to
control plane
Upon receiving a "leave" request
Increment
timers for all
Send IGMP
membership query
to all hosts
Timer exceeded
75s?
No
Yes
Yes
No
higher-resolution view of the state of the network but increases the burden on the management
network and the SDN controller. We choose an interval of 60 seconds which is about half of the
suggested value of 125 according to the RFC [22].
The rest of Figure 5 demonstrates the control plane logic for IGMP messages running for
each gateway. Whenever an end host reports joining or leaving a multicast group, the messages
are sent to the control plane logic for every gateway. If not already existent, the handling routine
sets up a “listeners_port” dictionary for each multicast group and adds an entry with the key being
the port number from which the report comes in and the value being an “IgmpListeners” class, as
shown in Figure 6. Then, the address of the listener will be added to the set “listeners_addresses”.
Whenever a “join” message arrives from any end host, the “time_counter” is reset to zero for the
listening port (“IgmpListeners”) it is attached to. Whenever a “leave” message arrives, it removes
such end host from the “listeners_addresses”. There are two cases where an “IgmpListeners” class
should terminate and evict itself, either when the timer reaches 75 or when all of its end hosts have
reported leaving the group.
Figure 6 Data structure for bookkeeping. The “_mcast” structure is a dictionary that has multicast addresses as the
keys and pointers as the values, pointing to other dictionaries that are “listeners_port”. Each “listeners_port”
dictionary has port numbers as keys and their values point towards “IgmpListeners” dictionaries.
The data structure in Figure 7 reflects what should be installed on the gateways. Each
existing entry in the “_mcast” dictionary corresponds to two action groups that are collections of
actions. They together have as many output ports as there are entries in the “listeners_port”
dictionary where those ports are split into two groups based on locality, one for southbound
interfaces and the other for northbound ones. The check for tunneling ID determines if a multicast
dict._mcast
224.0.0.
......
239.0.0.
......
......
dict.listeners_port
port 1
port 2
port 3
......
Class IgmpListeners
this_port_number
time_counter
listeners_addresses
add_listener()
del_listener()
count_up()
The third problem comes from dynamic configurability. OpenFlow is inherently incapable of
processing any tunneling protocol headers, so we rely upon OVS to provide the tunneling
endpoints. This requires reconfiguration for both the SDN controller and the gateways every time
we want to add a new grid into the system, which is in the order of 𝑂
With such consideration, we decided to switch to a design with the help of Multiprotocol
Label Switching (MPLS). In this design, we restrict the LAN switching to within the microgrids,
only allowing layer-3 traffic to go northbound. We incorporate proper ARP handling into the SDN
controller to reduce the amount of traffic flooding. In addition, because MPLS is native to
OpenFlow since version 1.1, we can reduce the northbound ports to only one per microgrid and
migrate the dynamic reconfiguration of power grid topology to the SDN controller, such as adding
a new microgrid to the existing setup. Now, the number of northbound links is reduced to the order
of 𝑂(𝑛) and the number of gateways requiring reconfiguration to zero.
Flow rules
Match: IP==224.0.0.1 Actions={(Forward to port 1,2,3), (Push MPLS), (Forward to port 101)}
......
......
......
......
Figure 8 Forwarding plane structure showing the flow table for multicast packets in MPLS design, which only
requires one entry per multicast group
Incoming traffic via the northbound interface is assumed to be always MPLS tagged, so it
is first stripped of its MPLS header and then checked to see if it is a multicast packet before being
sent to the multicast flow table. Another advantage in this design is that you do not need to
differentiate the ingress port of a packet because the same action can be applied to all without
causing broadcast storm, due to the fact that OpenFlow will ignore output actions to the ingress
interface, unless explicitly specified. For the same reason, we no longer need to split every flow
rule into two to separately handle traffic originating from the local grid versus traffic from remote
grids, and we completely get rid of the use of action groups, resulting in flow rules similar to those
in Figure 8. This design greatly simplifies the forwarding plane and we are down to only one entry
per multicast group on the forwarding plane.
So far, we have dealt only with the forwarding plane and control plane. Now we need to
introduce some human control on the fly. This is the application layer of the SDN hierarchy. We
incorporate some Representational State Transfer (REST) interfaces into our controller to achieve
some security measures as shown in Table 1.
Table 1 A List of APIs for the Application Layer
Name HTTP Request Path Method(s) Body of Response(s)
Description
list_table /gateways/table/{dpid} GET “ip to mac”
“mac_to_port”
Queries the controller for end hosts’ information at a specific datapath.
The response “ip_to_mac” shows the MAC addresses of end hosts with specific
IPs. “mac_to_port” shows which southbound ports the end hosts with specific
MAC addresses are attached to.
isolate /gateways/isolate/{dpid}/{port_no} POST “isolated_dpids_ports”
Shuts down a specific port of a specific Datapath (gateway). If the “port_no” is 0, it
shuts down every single port of the Datapath and cuts off any intra- and inter-grid
communication. If the “port_no” is the northbound port, it shuts down the inter-
grid communication but devices within the microgrid can still communicate with
each other properly. Both actions will eliminate other microgrids’ awareness of
multicast state of the isolated microgrid, which, in turn, eliminates unnecessary
northbound traffic that will eventually be rejected. If the “port_no” is one of the
southbound ports, only that specific device is isolated from the rest of the system.
The response “isolated_dpids_ports” keeps track of the isolated ports of the
gateways.
deisolate /gateways/deisolate/{dpid}/{port_no} POST “isolated_dpids_ports”
Undoes “isolate”.
The response is the same as that of the “isolate” API.
allocate /gateways/allocate/{dpid} POST None
Allocates resources for a new microgrid that joins the system.
The body of the POST method should contain the MPLS label assigned to this
microgrid, the port number of the northbound interface, the IP and MAC of the
default gateway and the subnet mask.
If the inputs are correct and the resources to allocate pose no conflicts with existing
setup, “200 OK” is returned. Otherwise it returns “400 Bad Request” or “
Conflict”.
