Download MULTICAST COMMUNICATION and more Lecture notes Network Design in PDF only on Docsity!
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Chapter 4 – Communication
Chapter 6 - Coordination
Wes J. Lloyd
School of Engineering
and Technology
University of Washington - Tacoma
TCSS 558: APPLIED DISTRIBUTED COMPUTING
Homework 2
Active Reading Quiz
Chapter 4 Communication
4.4 Multicast communication
Ch. 6 – Coordination
6.1 Clock synchronization
6.2 Logical clocks, Lamport clocks, Vector clocks
6.3 Distributed mutual exclusion
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. OBJECTIVES
How do you give MPI failure transparency?
Failure transparency involves hiding from the user, the fact that the system (or some aspect of it) has failed Providing failure transparency requires a system to implement fault tolerance Here is an FAQ on fault tolerance in OpenMPI: https://www.open-mpi.org/faq/?category=ft A number of techniques for fault tolerance have been employed previously in OpenMPI, but are not deprecated OpenMPI is said to mimic the fault tolerance provided by the FT-MPI framework March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. FEEDBACK – 3/
What types of messages are usually sent with gossip
spreading?
Gossip, in the context of Ch. 4.4, refers to multicast communication (one to many) across unstructured peer-to-peer network These are ad hoc connections where the structure of the network is unknown Multicast messages could be anything Multicast often concerns data dissemination – spreading data to many peer nodes as quickly and efficiently as possible March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. FEEDBACK - 2 A p ach e A ct i veMQ CH. 4.4: MULTICAST COMMUNICATION L14.
Sending data to multiple receivers
Many failed proposals for network-level / transport-level
protocols to support multicast communication
Problem: How to set up communication paths for
information dissemination?
Solutions: require huge management effort, human
intervention
Focus shifted more recently to peer-to-peer networks
Structured overlay networks can be setup easily and
provide efficient communication paths
Application-level multicasting techniques more successful
Gossip-based dissemination: unstructured p2p networks
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. MULTICAST COMMUNICATION
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Overlay network Virtual network implemented on top of an actual physical network Underlying network The actual physical network that implements the overlay March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. NETWORK STRUCTURE Broadcasting: every node in overlay receives message Key design issue: minimize the use of intermediate nodes for which the message is not intended Tree: if only the leaf nodes are to receive the multicast message, many intermediate nodes are involved Solution: construct an overlay network for each multicast group Flooding: each node simply forwards a message to each of its neighbors, except to the message originator March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. FLOOD-BASED MULTICASTING When no information on the structure of the overlay network Assume network can be represented as a Random graph Probability Pedge that two nodes are joined Overlay network will have: ½ * Pedge * N * (N-1) edges March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. RANDOM GRAPHS Random graphs allow us to assume some structure (# of nodes, # of edges) regarding the network by scaling the Pedge probability Assumptions may help then to reason or rationalize about the network…
….Washington state in winter?
When a node is flooding a message, concept is to enforce
a probability of message spread (pflood)
Throttles message flooding based on a probability
Implementation needs to consider # of neighbors to
achieve various pflood scores
With lower pflood messages may not reach all nodes
USEFULNESS: For random network with 10,000 nodes
With pedge = 0.1 and pflood =.
Achieves 50-fold reduction in messages vs. full flooding
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. PROBABILISTIC FLOODING For deterministic topologies (such as hypercube), design of efficient flooding scheme is much simpler If the overlay network is structured, this gives us a deterministic topology Schlosser et al [2002] – offer simple and efficient broadcasting scheme that relies on keeping track of neighbors per dimension March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. MESSAGE FLOODING Hypercube B roadcast N(1001) starts the network broadcast N(1001) neighbors {0001,1000,1011 ,1101} N(1001) Sends message to all neighbors Edge Labels (which bit is c hanged, 1st, 2 n d, 3rd, 4 th…) Edge to 0001 – labeled 1 – change the 1st bit Edge to 1000 – labeled 4 – change the 4th^ bit Edge to 1011 – labeled 3 – change the 3rd^ bit Edge to 1101 – labeled 2 – change the 2nd^ bit RULE: no des only forward along edges with a higher dimension Node 1101 receives message on edge labeled 2 Broadcast msg is only forwarded on higher dimension edges March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. MESSAGE FLOODING - 2
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Does not guarantee all nodes will be updated The fraction of nodes s, that remain susceptible grows relative to the probability that node P stops propagating when finding a node already having the message Fraction of nodes not updated remains < 0.20 with high pstop Susceptible nodes (s) vs. probability of stopping March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. RUMOR SPREADING - 2
Taking network topology into account can help
When gossiping, nodes connected to only a few other
nodes are more likely to be contacted
Epidemic protocols assume:
For gossiping, nodes are randomly selected
One node, can randomly select any other node in the
network
Complete set of nodes is known to each member
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. DIRECTIONAL GOSSIPING
Gossiping is good for spreading data
But how can data be removed from the system?
Idea is to issue “death certificates”
Act like data records, which are spread like data
When death certificate is received, data is deleted
Certificate is held to prevent data element from
reinitializing from gossip from other nodes
Death certificates time-out after expected time required
for data element to clear out of entire system
A few nodes maintain death certificates forever
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. REMOVING DATA For example: Node P keeps death certificates forever Item X is removed from the system Node P receives an update request for Item X, but als o holds the death certificate for Item X Node P will recirculate the death certificate across the network for Item X March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. DEATH CERTIFICATE EXAMPLE
6.1 Clock Synchronization
Physical clocks
Clock synchronization algorithms
6.2 Logical clocks
Lamport clocks
Vector clocks
6.3 Mutual exclusion
6.4 Election algorithms
6.6 Distributed event matching (light)
6.7 Gossip-based coordination (light)
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. CHAPTER 6 - COORDINATION How can processes synchronize and coordinate data? Process synchronization Coordinate cooperation to grant individual processes temporary access to shared resources (e.g. a file) Data synchronization Ensure two sets of data are the same (data replication) Coordination Goal is to manage interactions and dependencies between activities in the distributed system Encapsulates synchronization March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. CHAPTER 6 - COORDINATION
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Synchronization challenges begin with time:
How can we synchronize computers, so they all agree on
the time?
How do we measure and coordinate when things happen?
Fortunately, for synchronization in distributed systems, it
is often sufficient to only agree on a relative ordering of
events
E.g. not actual time
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. COORDINATION - 2
Groups of processes often appoint a coordinator
Election algorithms can help elect a leader
Synchronizing access to a shared resource is achieved
with distributed mutual exclusion algorithms
Also in chapter 6:
Matching subscriptions to publications in publish-
subscribe systems
Gossip-based coordinate problems:
Aggregation Peer sampling Overlay construction March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. COORDINATION - 3 CH. 6.1: CLOCK SYNCHRONIZATION L14. Example: “make” is used to compile source files into binary object and executable files As an optimization, make only compiles files when the “last modified time” of source files is more recent that object and executables Consider if files are on a shared disk of a distributed system where there is no agreement on time Consider if the program has 1 ,000 source files March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. CLOCK SYNCHORNIZATION Updates from different machines, may have clocks set to different times Make becomes confused with which files to recompile March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. TIME SYNCHRONIZATION PROBLEM FOR DISTRIBUTED SYSTEMS March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. PHYSICAL CLOCKS Co mputer timers: precisely machined quartz crystals When under tension, they oscillate at a well defined frequency In analog electronics/communications crystals once used to set the frequency of two-way radio transceivers for Today, crystals are associated with a counter and holding register on a digital computer. Each oscillation decrements a counter by one When counter gets to zero, an interrupt fires Can program timer to generate interrupt, let’s say 60 times a second, or another frequency to track time 1960s ERA radio crystal
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Must estimate network delays when synchronizing with remote UTC receiver clocks / time servers Time server B Client A
- A sends message to B, with timestamp T
- B records time of receipt T2 (from local clock)
- B returns response with send time T3, and receipt time T
- A records arrival of T Assuming propagation delay of ABA is the same Estimate propagation delay: Add delay to time March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. NTP - 2 Cannot set clocks backwards (recall “make” file example) Instead, temporarily slow the progress of time to allow fast clock to align with actual time Change rate of clock interrupt routine Slow progress of time until synchronized NTP accuracy is within 1-50ms In Ubuntu Linux, to quickly synchronize time: $apt install ntp ntpdate Specify local timeservers in /etc/ntp.conf server time.u.washington.edu iburst server bigben.cac.washington.edu iburst Shutdown service (sudo service ntp stop) Run ntpdate: (sudo ntpdate time.u.washington.edu) Startup service (sudo service ntp start) March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. NTP - 3
Berkeley time daemon server actively polls network to
determine average time across servers
Suitable when no machine has a UTC receiver
Time daemon instructs servers how much to adjust clocks
to achieve precision
Accuracy can not be guaranteed
Berkeley is an internal clock synchronization algorithm
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. BERKELEY ALGORITHM Sensor networks bring unique challenges for clock synchronization Address resource constraints: limited power, multihop routing slow Re ference broadcast s y nchronization (RB S) Provides precision of time, not accuracy as in Berkeley No UTC clock available RBS sender broadcasts a reference message to allow receivers to adjust clocks No multi-hop routing Time to propagate a signal to nodes is roughly constant Message propagation time does not consider time spent waiting in NIC for message to send Wireless network resource contention may force wait before message even ca n be sent March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. CLOCK SYNCHRONIZATION IN WIRELESS NETWORKS Node broadcasts reference message m Each node p records time Tp,m when m is received Tp,m is read from node p’s clock Two nodes p and q can exchange delivery times to estimate mutual relative offset Then calculate relative average offset for the network: Where M is the total number of reference messages sent Nodes can simply store offsets instead of frequently synchronizing clocks to save energy March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. REFERENCE BROADCAST SYNCHRONIZATION (RBS) Cloud skew: over time clocks drift apart Averages become less precise Elson et al. propose using standard linear regression to predict offsets, rather than calculating them IDEA: Use node’s history of message times in a simple linear regression to continuously refine a formula with coefficients to predict time offsets: March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. REFERENCE BROADCAST SYNCHRONIZATION (RBS) - 2
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
CH. 6.2: LOGICAL CLOCKS L14. In distributed systems, synchronizing to actual time may not be required… It may be sufficient for every node to simply agree on a current time (e.g. logical) L o gica l cl o cks provide a mechanism for capturing chronological and cau sal relationships in a distributed system Think co u nter s... Leslie Lamport [1978] seminal paper showed that absolute clock synchronization often is not required Processes simply need to agree on the order in which events occur March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. LOGICAL CLOCKS Happens-before relation AB: Event A, happens before event B… All processes must agree that event A occurs first Then afterward, event B Actual time not important... If event A is the event of proc P1 sending a msg to a proc P2, and event B is the event of proc P2 receiving the msg, then AB is also true... The assumption here is that message delivery takes time Happens before is a transitive relation: AB, BC, therefore AC March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. LOGICAL CLOCKS - 2 If two events, say event X and event Y do not exchange messages, not even via third parties, then the sequence of XY vs. YX can not be determined!! Within the system, these events appear concurrent Co ncurrent: nothing can be said about when the events happened, or which event occurred first Clock time, C, must always go forward (increasing), never backward (decreasing) Corrections to time can be made by adding a positive value, but never by subtracting one March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. LOGICAL CLOCKS – 3 Three processes each with local clocks Lampor t’s algorithm corrects their values March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. LOGICAL CLOCKS - 4 Events: 6: P1 send m1 to P 16: P2 receives m 24: P2 sends m2 to P 40: P3 receives m 60: P3 sends m3 to P 56: P2 receives m 5 6: P2 clock reset= 64: P2 sends m4 to P 54: P1 receives m 7 0: P1 clock reset= March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. LOGICAL CLOCKS
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Consider the messages: P2 receives m1 , and subsequently sends m Causality: Sending m3 may depend on what’s contained in m P2 receives m2, receiving m2 is not related to receiving m Is s ending m3 causally dependent on receiving m2? March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. WHAT IS CAUSALITY? Vector clocks keep track of causal his tory If two local events happened at process P, then the causal history H(p2) of event p2 is {p1,p2} P sends messages to Q (event p3) Q previously performed event q Q records arrival of message as q Causal histories merged at Q H(q2)= {p1,p2,p3,q1,q2} Fortunately, can simply store history of last event, as a vector clock H(q2) = (3,2) Each entry corresponds to the last event at the process March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. VECTOR CLOCKS Each process maintains a vector clock which Captures number of events at the local process (e.g. logical clock) Captures number of events at all other processes Causality is captured by: For each event at Pi, the vector clock (VCi) is incremented The msg is timestamped with VCi; and sending the msg is recorded as a new event at Pi Pj adjusts its VCj choosing the max of: the message timestamp –or- the local vector clock (VCj) March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. VECTOR CLOCKS - 2 P 1 P 2 (1,0) (2,0) (3,0) (0,1) (3,2) m 1 Pj knows the # of events at Pi based on the timestamps of the received message Pj learns how many events have occurred at other processes based on timestamps in the vector These events “may be causally dependent“ In other words: they may have been necessary for the message(s) to be sent… March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. VECTOR CLOCKS - 3 Local clock is underlined March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. VECTOR CLOCKS EXAMPLE ts (m 2 ) ts(m 4 ) ts(m 2 )<ts(m 4 ) ts(m 2 )>ts(m 4 ) Conclusion (2,1,0) (4,3,0) Yes No m2 may causally precede m
CAUSALITY
P3 can’t determine if m4 may be causally dependent on m Is m4 causally dependent on m3? March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. VECTOR CLOCKS EXAMPLE - 2 ts (m 2 ) ts(m 4 ) ts(m 2 )<ts(m 4 ) ts(m 2 )>ts(m 4 ) Conclusion (4,1,0) (2,3,0) No No m2 and m4 may conflict
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Disclaimer: Without knowing actual information contained in messages, it is not possible to state with certainty that there is a causal relationship or perhaps a conflict Vector clocks can help us suggest possible causality We never know for sure… March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. VECTOR CLOCKS - 4 CH. 6.3: DISTRIBUTED MUTUAL EXCLUSION L14.
Coordinating access among distributed processes to a
shared resource requires Distributed Mutual Exclusion
Token-based algorithms:
Mutual exclusion by passing a “token” between nodes
Nodes often organized in ring
Only one token, holder has access to shared resource
Avoids starvation: ever yone gets a chance to obtain lock
Avoids deadlock: easy to avoid
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. DISTRIBUTED MUTUAL EXCLUSION Construct overlay network Establish logical ring among nodes Single token circulated around the nodes of the network Node having token can access shared resource If no node accesses resource, token is constantly circulated around ring March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. TOKEN-RING ALGORITHM
1. If token is lost, token must be regenerated
Problem: may accidentally circulate multiple tokens
2. Hard to determine if token is lost
What is the difference between token being lost and a
node holding the token for a long time?
3. When node crashes, circular network route is broken
Dead nodes can be detected by adding a receipt message
for when the token passes from node-to-node
When no receipt is received, node assumed dead
Dead process can be “jumped” in the ring
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. TOKEN-RING CHALLENGES Permission-based algorithms Processes must require permission from other processes before first acquiring access to the resource Centralized algorithm Elect a single leader node to coordinate access to shared resource(s) Manage mutual exclusion on a distributed system similar to how it mutual exclusion is managed for a single system Nodes must all interact with leader to obtain “the lock” March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14. DISTRIBUTED MUTUAL EXCLUSION - 2
[Winter 2019] School of Engineering and Technology,
UW-Tacoma
Problem: Multicast communication required –or- each node must maintain full group membership Track nodes entering, leaving, crashing… Problem: Every process is involved in reaching an agreement to grant access to a shared resource This approach may not scale on resource-constrained systems Solution: Can relax total agreement requirement and proceed when a s imple majority of nodes grant permission Presumably any one node locking the resource prevents agreement Distributed algorithm for mutual exclusion works best for: Small groups of processes When memberships rarely change March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14.
CHALLENGES WITH
DISTRIBUTED ALGORITHM - 2 QUESTIONS
March 6, 2019 TCSS558: Applied Distributed Computing [Winter 2019] School of Engineering and Technology, University of Washington - Tacoma L14.
EXTRA SLIDES
75
