CA1223372A - Fault-tolerant atomic broadcast methods - Google Patents
Fault-tolerant atomic broadcast methodsInfo
- Publication number
- CA1223372A CA1223372A CA000482185A CA482185A CA1223372A CA 1223372 A CA1223372 A CA 1223372A CA 000482185 A CA000482185 A CA 000482185A CA 482185 A CA482185 A CA 482185A CA 1223372 A CA1223372 A CA 1223372A
- Authority
- CA
- Canada
- Prior art keywords
- message
- processor
- processors
- network
- fault
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
- G06F15/17356—Indirect interconnection networks
- G06F15/17368—Indirect interconnection networks non hierarchical topologies
- G06F15/17381—Two dimensional, e.g. mesh, torus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
Abstract
FAULT-TOLERANT ATOMIC BROADCAST METHODS
Abstract of the Invention A method for reliably broadcasting information in a point-to-point network of processors in the presence of component faults provided that the network remains connnected using only an exchange of messages. The method possesses the properties (1) that every message broadcast by a fault-free processor is accepted exactly once by all fault-free processors within a bounded time, (2) that every message broadcast is either accepted by all fault-free processors or none of them, and (3) that all messages accepted by fault-free processors are accepted in the same order by all those processors.
The method is based on a diffusion technique for broadcasting information and on special message validity tests for tolerating any number of component failures up to network partitioning or successful forgery.
Abstract of the Invention A method for reliably broadcasting information in a point-to-point network of processors in the presence of component faults provided that the network remains connnected using only an exchange of messages. The method possesses the properties (1) that every message broadcast by a fault-free processor is accepted exactly once by all fault-free processors within a bounded time, (2) that every message broadcast is either accepted by all fault-free processors or none of them, and (3) that all messages accepted by fault-free processors are accepted in the same order by all those processors.
The method is based on a diffusion technique for broadcasting information and on special message validity tests for tolerating any number of component failures up to network partitioning or successful forgery.
Description
37~ .
FAULT-TOLERANT ATOMIC BROADCAST METHODS
Technical Field This invention relates to the atomic broadcastinq of information in a distributed computing net~ork in the presence of faults, where the processors in the network exchange messages asynchronously.
Background of the Invention Methods for reliably broadcasting information in a distributed network of processors enable the fault-free components of that system to have consistent views of the global system state in the presence of faulty components.
A reliable broadcast method guarantees (1) that a message broadcast is either accepted by all fault-free processors exactly once or by none of them (atomicity), (2) that a message broadcast by a fault-free sender is accepted correctly by all fault-free processors after a known time (termination), and (3) that all messages accepted by fault-free processors are accepted in the same order by all those processors (order). Such a method is an essential part of many protocols for distributed systems, e.g. updating replicated data.
Dolev and Strong, "Authenticated Methods for Byzantine Agreement", SIAM Journal of Computing, Vol. 12, No. 4, November 1983, describe a method for achieving atomic broadcast assuming a (logically) fully-connected network of processors. In order to tolerate F faults, their method runs for (F+l) phases and exchanges (N-1)2 logical messages, where N is the number of processors in the network and the duration of a phase is the maximum network transmission delay.
- SA9-84-017 -l-337~
Reference should also be made to Dolev et al "A
Me'hod for Achieving Multiple Process 3r Agre~ ent Optimized for No Faults", United States Patent `~o.
't,-~,`3,015, issued to ~atent on February ~, 1986. ~hey describe a met~od for atomic broadcast in a r~liable (E'+1) cor.;iected network of N processors with ~laranteed early stoppiny in the absence of faults, and eventual stoppins for F < (N/2) faults.
The Invention It is the object of this invention to devise a method for reliably broadcasting information in a point-to-point network of processors in the presence of component faults, provided the network remains connected, using exchange of messages. Any such method must possess the following properties: (1) every message broadcast by a fault-free processor is accepted exactly once by all fault-free processors within a bounded time (termination), (~) every messa~e broadcast is either accepted by all fault-free processors or none of them (atomicity), and (3) all messages accepted by fault-free processors are accepted in the same order by all those processors (order).
The objects are satisfied by a machine-implementa~le method in which each processor maintains an amendable ~5 his~ory of broadcast messages, and executes the following steps:
(a) in response to a message broadcast request local to any processor s, processor s assigns to that message a network unique identifier (IDE~TIFIER) and a timestamp (TIMESTAMP), adds the message to its local history, signs the message, including the assigned values, and broadcasts the result to its neighbors (two processors are neighbors if they are connected ~y a direct communication link);
(b) in response to the receipt of a signed message X, each processor p (l) verifies its authenticity and derives from it the original message M and a sequence of signatures of the processors that have signed the message ~in the order that they have signed the message with sender's signature being the first signature); and (2) ascertains messaqe validity as to its not being already signed by p, as to its not having duplicate signatures, as to its being nonduplicative of a previously received message, and as to its timeliness; and (c) each processor q within the subset of processors p finding the message M valid performs the further steps of: (1) updatin~ its message history by adding the received message M thereto; (2) signing the received message X and broadcasting the result to its network adjacent neighbors except the processor from whom the message X was received; and (3) after a predetermined amount of time from the time of message origination (whose value depends on the network topology, communication delay, and maximum clock deviation) accepting the message M if it is determined to be valid, and erasing that message from the history.
The following consequences result when the method of this invention is invoked:
(a) If a processor s initiates a broadcast of a message at local clock time t then: (1) if processor s is fault free then every processor p that is fault free during the time interval (t,t+BYZT), where BYZT is a constant, must accept the message by time t~BYZT
(termination); and (2) if p and ~ are fault free during the time interval ~t,t+BYZT), then by time t+BYZT either both accept the same message or none of them accepts the message (atomicity).
(b) The messages that are accepted by fault-free processors are accepted in the same order by all those processors (order).
(c) The termination time BYZT is given by BYZT = F x (TDEL + DMAX) + D~AX + (DIAM(F) x TDEL) where TDEL is the maximum network transmission delay, DMAX is the maximum clock deviation between every pair of processors, and DIAM(F) is the worst case network diameter after F faults.
(d) In the absence of faults, the number of messages sent by this method to perform one broadcast is given by
FAULT-TOLERANT ATOMIC BROADCAST METHODS
Technical Field This invention relates to the atomic broadcastinq of information in a distributed computing net~ork in the presence of faults, where the processors in the network exchange messages asynchronously.
Background of the Invention Methods for reliably broadcasting information in a distributed network of processors enable the fault-free components of that system to have consistent views of the global system state in the presence of faulty components.
A reliable broadcast method guarantees (1) that a message broadcast is either accepted by all fault-free processors exactly once or by none of them (atomicity), (2) that a message broadcast by a fault-free sender is accepted correctly by all fault-free processors after a known time (termination), and (3) that all messages accepted by fault-free processors are accepted in the same order by all those processors (order). Such a method is an essential part of many protocols for distributed systems, e.g. updating replicated data.
Dolev and Strong, "Authenticated Methods for Byzantine Agreement", SIAM Journal of Computing, Vol. 12, No. 4, November 1983, describe a method for achieving atomic broadcast assuming a (logically) fully-connected network of processors. In order to tolerate F faults, their method runs for (F+l) phases and exchanges (N-1)2 logical messages, where N is the number of processors in the network and the duration of a phase is the maximum network transmission delay.
- SA9-84-017 -l-337~
Reference should also be made to Dolev et al "A
Me'hod for Achieving Multiple Process 3r Agre~ ent Optimized for No Faults", United States Patent `~o.
't,-~,`3,015, issued to ~atent on February ~, 1986. ~hey describe a met~od for atomic broadcast in a r~liable (E'+1) cor.;iected network of N processors with ~laranteed early stoppiny in the absence of faults, and eventual stoppins for F < (N/2) faults.
The Invention It is the object of this invention to devise a method for reliably broadcasting information in a point-to-point network of processors in the presence of component faults, provided the network remains connected, using exchange of messages. Any such method must possess the following properties: (1) every message broadcast by a fault-free processor is accepted exactly once by all fault-free processors within a bounded time (termination), (~) every messa~e broadcast is either accepted by all fault-free processors or none of them (atomicity), and (3) all messages accepted by fault-free processors are accepted in the same order by all those processors (order).
The objects are satisfied by a machine-implementa~le method in which each processor maintains an amendable ~5 his~ory of broadcast messages, and executes the following steps:
(a) in response to a message broadcast request local to any processor s, processor s assigns to that message a network unique identifier (IDE~TIFIER) and a timestamp (TIMESTAMP), adds the message to its local history, signs the message, including the assigned values, and broadcasts the result to its neighbors (two processors are neighbors if they are connected ~y a direct communication link);
(b) in response to the receipt of a signed message X, each processor p (l) verifies its authenticity and derives from it the original message M and a sequence of signatures of the processors that have signed the message ~in the order that they have signed the message with sender's signature being the first signature); and (2) ascertains messaqe validity as to its not being already signed by p, as to its not having duplicate signatures, as to its being nonduplicative of a previously received message, and as to its timeliness; and (c) each processor q within the subset of processors p finding the message M valid performs the further steps of: (1) updatin~ its message history by adding the received message M thereto; (2) signing the received message X and broadcasting the result to its network adjacent neighbors except the processor from whom the message X was received; and (3) after a predetermined amount of time from the time of message origination (whose value depends on the network topology, communication delay, and maximum clock deviation) accepting the message M if it is determined to be valid, and erasing that message from the history.
The following consequences result when the method of this invention is invoked:
(a) If a processor s initiates a broadcast of a message at local clock time t then: (1) if processor s is fault free then every processor p that is fault free during the time interval (t,t+BYZT), where BYZT is a constant, must accept the message by time t~BYZT
(termination); and (2) if p and ~ are fault free during the time interval ~t,t+BYZT), then by time t+BYZT either both accept the same message or none of them accepts the message (atomicity).
(b) The messages that are accepted by fault-free processors are accepted in the same order by all those processors (order).
(c) The termination time BYZT is given by BYZT = F x (TDEL + DMAX) + D~AX + (DIAM(F) x TDEL) where TDEL is the maximum network transmission delay, DMAX is the maximum clock deviation between every pair of processors, and DIAM(F) is the worst case network diameter after F faults.
(d) In the absence of faults, the number of messages sent by this method to perform one broadcast is given by
2 x L - (N-l) where N is the number of processors, and L is the number of communication links~
Fundamentally, the method of this invention is based 2 on a diffusion technique for broadcasting information, and on special message validity tests for tolerating any number of component failures up to network partitioning or successful forgery. This method is an advance over methods which simply adopt a diffusion technique for 25 broadcasting in that prior art methods can only tolerate a small subset of the fault class that the method of this invention can tolerate. Further, the method of this invention is also more efficient ~han the one published by ., t~t~
Dolev and Strong in that fewer messages and less time are needed to complete a broadcast.
In order to facilitate appreciation of this invention, it is helpful to consider faults as being of several classes: (1) a fault that causes a component not to respond to a given service req~lest is called an "omission" fault; (2) a fault that causes a component to respond to a given request either too early or too late is called a "timing" fault; and (3) a fault that causes a component to deliver a different response than expected is called a "Byzantine" fault. That is, the set of faults covered by each class is a superset of the one covered by the preceding class, i.e. class 3 includes class 2, and class 2 includes class 1.
In this invention, a method for reliably broadcasting information in a network of message exchanging processors with approximately synchronized clocks is presented which is tolerant of the most general fault class, i.e. the "Byzantine" aults, and remains operable so long as the network is not disconnected by faulty components. Furthermore, it is demonstrated that if the faults to be tolerated are restricted to the less general classes, i.e. "timing" or "omissîon" faults, then simpler methods can be derived from the general case.
Brief Description of the Figures of the Drawing .
Figures 1, 2, 3, and 4 exhibit a high-level decision flow of the three tasks comprising the inventive method of which Figure 1 depicts the SEND task, Figures 2 and 3 depict the RECEIVE task, and Figure 4 depicts the END
task.
.
33~72 Eigures 5, 6, and 7 contain a pseudocode implementation of the respective SEND, RECEIV~, and ~ND
tasks.
Figure 8 depicts a network of processors used in Examples 1 and 2.
Figures 9, 10, and 11 exhibit a pseudocode implementation of respective SEND, RECEIVE, and END tasks of the method of this invention tolerating only timing faults.
Figures 12, 13, and 14 exhibit a pseudocode implementation of respective SEND, RECEIVE, and END tasks of the method of this invention tolerating only omission faults.
Figures 15, 16, and 17 exhibit a pseudocode implementation of respective SEND, RECEIVE, and END tasks of the method of this invention tolerating only omission faults without need of clock synchronization and without guaranty of message order.
Description of the Preferred Embodiment and Industrial Applicabilitv Conditions and Notations .
The method presented in this invention is operable in an environment in which:
(a) The processors are connected by a point-to-point communication network.
~ b) The clocks of the fault-free processors are synchronized up to a known deviation. That is, the readings of any pair of clocks at any given time cannot :, ~ _ _ _ .. . . ..
r--~2~37~
differ by more than a known value. This assumption is dropped for a variant of the method that only tolerates "omission" faults. Such a clock sy~chroni~ation m~thod is presented by Halpern et ~l, in United States Patent No. 4,531,185, issued July~23, 19~5.
(c) If two processors and the link joininy them are not faulty, then there exists a known upper bound on the time required for a message to be prepared by the sender, transmitted over the link, and processed a~ the receiving end.
(d) The processors are numbered 1, 2, ..., N. If processor i intends to send a message m, there is an encoding function ei such that (l) no processor other than i can generate the message ei(m), i.e. no message can be forged; and (2) if processor j receives ei(m), it can decode m and determine that i was the processor that sent the message, i.e. the message can be authenkicated. Such an encoding/decoding method is described by Rivest et al, "A Method for Obtaining Digital Signatures and Public-key Cryptosystems", Communications of the ACM, Vol. 21, pp. 120-126, 1978.
In describing the method of this invention, the following notations will be used:
?5 F The number OI faults to be tolerated.
This is a parameter of the method, and counts both processor and link failures.
DIAM(F) The worst case diameter of the network after the occurrence of F
faults.
~ ~;233~2 TDEL The worst case transmission and processing delay from one processor to its neighboring processor over one link. This interval extends from the time the first processor sends the message until the time the receiving processor has processed the message and is ready to send any re~lired response. When messages are sent to all neighbors of a processor, TDEL represents the worst case time from the time the first message is sent until the time the last neighbor has processed the message.
DMAX The worst case deviation of the clocks of the correctly operating processors.
BYZT The time required to complete the atomic broadcast:
BYZT = F x (TDEL + DMAX) +
DMAX ~ DIAM(F) x TDEL.
IDENTIFIER A globally unigue identifier for a particular broadcast.
.
TIMESTAMP An indication of the local clock time at which the broadcast message was prepared. That is, given a TIMEST~MP
one can unambiguously determine the clock that was current at the time of that TIMESTAMP.
' :
VALUE The conteilts of the message to be broadcast.
MESSAGE A message is a tuple of the form (IDENTIFIER,TIMESTAMP,VALUE) HISTORY A set of tuples of the form (MESSAGE,NUMBER OF VALUES) where NUMBER OF VALUES is an indicator stored with a message in HISTORY which reflects whether more than one VALUE has been associated authentically with the given MESSAGE. (If the NUMBER OF VALUES is 2, then the originator of the message is faulty or the authentication protocol has been compromised.) Observe that given an IDENTIFIER, the associated message can be uniquely located in HISTQRY. New tuples may be added to HISTORY, and stored tuples may be deleted from it.
N The number of processors in the network.
L The number of communication links.
~2~372 he Local Node Exec;ltion Environrnent As a preliminary, the method of this invenl on is exec~ttable upon a computinc3 system of the -_ype including one or more CPU's, each having a main store, input~outpu channel, control unit, di~ect access sto~age devices, local addressable clocks, and other I,/O devices coupled thereto. Such a system is described in Amdahl el al, USP 3,400,371, "Data Processing Syste~", issued September 3, 1968. The Amdahl system includes 2S a resource all of the facilities of either the co~puting system of an operating system running the-eon which are required for the execution of a process including the method of this invention. Typical resources include a main store, I/O devices, the CPU, data sets, interval timer, and control or processing programs. Furthermore, such systems are capable of "multiprograrlming". This pertains to the concurrent execution of two or more processes by a computing system, and can be managed on a computer running under an IBM System/370 operating syste~, 20 as described in IBM Publication GC28-6646, July 1973, and listed in IBM System/360 bibliography, GA22-6822.
Networks of asynchronously operating CPU's are described in Fitzgerald, USP 3,940,743, "Interconnecting Unit for Independently Operable Data Processing Systems", issued 25 February 24, 1975; and Antonaccio et al, USP 4,223,380, "Distributed Multiprocessor Communication System", issued September 16, 1980.
Elow Diagram Depiction of Method Execution and Its Task Organization Referring now to Figures 1, 2, 3, and 4, there is exhibited a high-level decision flow of the inventive method. Residing at each processor in the network are three tasks that are running concurrently: the SEND task, the RECEIVE task, and the END task.
* Trademark
Fundamentally, the method of this invention is based 2 on a diffusion technique for broadcasting information, and on special message validity tests for tolerating any number of component failures up to network partitioning or successful forgery. This method is an advance over methods which simply adopt a diffusion technique for 25 broadcasting in that prior art methods can only tolerate a small subset of the fault class that the method of this invention can tolerate. Further, the method of this invention is also more efficient ~han the one published by ., t~t~
Dolev and Strong in that fewer messages and less time are needed to complete a broadcast.
In order to facilitate appreciation of this invention, it is helpful to consider faults as being of several classes: (1) a fault that causes a component not to respond to a given service req~lest is called an "omission" fault; (2) a fault that causes a component to respond to a given request either too early or too late is called a "timing" fault; and (3) a fault that causes a component to deliver a different response than expected is called a "Byzantine" fault. That is, the set of faults covered by each class is a superset of the one covered by the preceding class, i.e. class 3 includes class 2, and class 2 includes class 1.
In this invention, a method for reliably broadcasting information in a network of message exchanging processors with approximately synchronized clocks is presented which is tolerant of the most general fault class, i.e. the "Byzantine" aults, and remains operable so long as the network is not disconnected by faulty components. Furthermore, it is demonstrated that if the faults to be tolerated are restricted to the less general classes, i.e. "timing" or "omissîon" faults, then simpler methods can be derived from the general case.
Brief Description of the Figures of the Drawing .
Figures 1, 2, 3, and 4 exhibit a high-level decision flow of the three tasks comprising the inventive method of which Figure 1 depicts the SEND task, Figures 2 and 3 depict the RECEIVE task, and Figure 4 depicts the END
task.
.
33~72 Eigures 5, 6, and 7 contain a pseudocode implementation of the respective SEND, RECEIV~, and ~ND
tasks.
Figure 8 depicts a network of processors used in Examples 1 and 2.
Figures 9, 10, and 11 exhibit a pseudocode implementation of respective SEND, RECEIVE, and END tasks of the method of this invention tolerating only timing faults.
Figures 12, 13, and 14 exhibit a pseudocode implementation of respective SEND, RECEIVE, and END tasks of the method of this invention tolerating only omission faults.
Figures 15, 16, and 17 exhibit a pseudocode implementation of respective SEND, RECEIVE, and END tasks of the method of this invention tolerating only omission faults without need of clock synchronization and without guaranty of message order.
Description of the Preferred Embodiment and Industrial Applicabilitv Conditions and Notations .
The method presented in this invention is operable in an environment in which:
(a) The processors are connected by a point-to-point communication network.
~ b) The clocks of the fault-free processors are synchronized up to a known deviation. That is, the readings of any pair of clocks at any given time cannot :, ~ _ _ _ .. . . ..
r--~2~37~
differ by more than a known value. This assumption is dropped for a variant of the method that only tolerates "omission" faults. Such a clock sy~chroni~ation m~thod is presented by Halpern et ~l, in United States Patent No. 4,531,185, issued July~23, 19~5.
(c) If two processors and the link joininy them are not faulty, then there exists a known upper bound on the time required for a message to be prepared by the sender, transmitted over the link, and processed a~ the receiving end.
(d) The processors are numbered 1, 2, ..., N. If processor i intends to send a message m, there is an encoding function ei such that (l) no processor other than i can generate the message ei(m), i.e. no message can be forged; and (2) if processor j receives ei(m), it can decode m and determine that i was the processor that sent the message, i.e. the message can be authenkicated. Such an encoding/decoding method is described by Rivest et al, "A Method for Obtaining Digital Signatures and Public-key Cryptosystems", Communications of the ACM, Vol. 21, pp. 120-126, 1978.
In describing the method of this invention, the following notations will be used:
?5 F The number OI faults to be tolerated.
This is a parameter of the method, and counts both processor and link failures.
DIAM(F) The worst case diameter of the network after the occurrence of F
faults.
~ ~;233~2 TDEL The worst case transmission and processing delay from one processor to its neighboring processor over one link. This interval extends from the time the first processor sends the message until the time the receiving processor has processed the message and is ready to send any re~lired response. When messages are sent to all neighbors of a processor, TDEL represents the worst case time from the time the first message is sent until the time the last neighbor has processed the message.
DMAX The worst case deviation of the clocks of the correctly operating processors.
BYZT The time required to complete the atomic broadcast:
BYZT = F x (TDEL + DMAX) +
DMAX ~ DIAM(F) x TDEL.
IDENTIFIER A globally unigue identifier for a particular broadcast.
.
TIMESTAMP An indication of the local clock time at which the broadcast message was prepared. That is, given a TIMEST~MP
one can unambiguously determine the clock that was current at the time of that TIMESTAMP.
' :
VALUE The conteilts of the message to be broadcast.
MESSAGE A message is a tuple of the form (IDENTIFIER,TIMESTAMP,VALUE) HISTORY A set of tuples of the form (MESSAGE,NUMBER OF VALUES) where NUMBER OF VALUES is an indicator stored with a message in HISTORY which reflects whether more than one VALUE has been associated authentically with the given MESSAGE. (If the NUMBER OF VALUES is 2, then the originator of the message is faulty or the authentication protocol has been compromised.) Observe that given an IDENTIFIER, the associated message can be uniquely located in HISTQRY. New tuples may be added to HISTORY, and stored tuples may be deleted from it.
N The number of processors in the network.
L The number of communication links.
~2~372 he Local Node Exec;ltion Environrnent As a preliminary, the method of this invenl on is exec~ttable upon a computinc3 system of the -_ype including one or more CPU's, each having a main store, input~outpu channel, control unit, di~ect access sto~age devices, local addressable clocks, and other I,/O devices coupled thereto. Such a system is described in Amdahl el al, USP 3,400,371, "Data Processing Syste~", issued September 3, 1968. The Amdahl system includes 2S a resource all of the facilities of either the co~puting system of an operating system running the-eon which are required for the execution of a process including the method of this invention. Typical resources include a main store, I/O devices, the CPU, data sets, interval timer, and control or processing programs. Furthermore, such systems are capable of "multiprograrlming". This pertains to the concurrent execution of two or more processes by a computing system, and can be managed on a computer running under an IBM System/370 operating syste~, 20 as described in IBM Publication GC28-6646, July 1973, and listed in IBM System/360 bibliography, GA22-6822.
Networks of asynchronously operating CPU's are described in Fitzgerald, USP 3,940,743, "Interconnecting Unit for Independently Operable Data Processing Systems", issued 25 February 24, 1975; and Antonaccio et al, USP 4,223,380, "Distributed Multiprocessor Communication System", issued September 16, 1980.
Elow Diagram Depiction of Method Execution and Its Task Organization Referring now to Figures 1, 2, 3, and 4, there is exhibited a high-level decision flow of the inventive method. Residing at each processor in the network are three tasks that are running concurrently: the SEND task, the RECEIVE task, and the END task.
* Trademark
3~7~
The SEND task on each processor s is responsible for initiating the broadcast of messages. Upon receipt of a ~roadcast request, the SEND task executes the following steps of: (1) generating an IDENTIFIER and a TIMESTAMP
5 for that message, to produce the MESSAGE
(IDENTIFIER,TIMESTAMP,VALUE);
~2) storing the tuple <MESSAGE,1> in its local HISTORY;
(3) setting its local interval timer to schedule the END
task at TIMESTAMP+BYZT to accept the MESSAGE VALUE and removing the MESSAGE from HISTORY (observe that by the time TIMESTAMP+BYZT the message should have been received by all fault-free processors); and finally (4) signing the MESSAGE and broadcasting the result to all of its neighbors.
The RECEIVE task on each processor is constantly waiting to receive messages. The main responsibility of the RECEIVE task is to filter out invalid messages, and store each valid message in local HISTORY the first time it is received along with a count of the number of diferent values that are received with that message IDENTIFIER. Observe that a fault-free sender associates an IDENTIFIER with only one message value, whereas a faulty sender may associate several message values with the same IDENTIFIER. It is only the former kind that must be accepted, while the latter must be discarded.
.
Specifically, upon receipt of a signed message X
from the network by a processor s, the RECEIVE task on that processor verifies its authenticity and derives from it the original MESSAGE and a sequence of signatures of the processors that have signed that message (in the order that they have signed that message with sender's signature being the first signature). If the message is determined to be "forged", it will be discarded. If the message is determined to be "authentic", then the RECEIVE
task will execute the following tests: (1) whether the MESSAGE has duplicate signatures; (2) whether the MESSAGE
has more signatures than F~DIAM(F), i.e. too many hops;
(3) whether the MESSAGE has arrived too early, possibly due to some clock failure; and (4) whether the MESSAGE has arrived too late, possibly due to timing faults. If the answer to any one of the above tests is "yes", then the received message is determined to be invalid and is, therefore, discarded.
Once the message passes all of the above tests, then the RECEIVE task checks if it has already seen this message by consulting its HISTORY. That is, the HISTORY
is searched for a tuple (M,NUMBER OF VALUES) such that M's IDENTIFIER is the same as MESSAGE's IDENTIFIER. The search results in one of the following outcomes:
(a) No such tuple is found; this signifies that this MESSAGE is received for the first time and has not been seen before. The RECEIVE task then executes the following steps of: (1) storing the tuple (ME~SAGE,l) in the HISTORY to record the fact that MESSAGE has been received already once and has only one value associated with it; (2) setting its local interval timer to schedule the END task at time TIMESTAMP+BYZT to process the MESSAGE
(at that time, the END task will remove the MESSAGE from local HISTORY, and accepts the message value there is only one value associated with the MESSAGE); and finally (3) signing the received (signed) message X and broadcasting the result to all o its neighbors except the one from whom the message X was received.
(b) A tuple (M,l) is found such that its associated VALUE is the same as the VALUE associated with the ~3'7~
received MESSAGE. This represents the case where another copy of the same message is received again, and may happen since several copies of a message travel the network over different paths and reach the same processor at different times. Since the received MESS~GE has already been seen, it will be discarded.
(c) A tuple (M,l) is found such that its associated value is not the same as the VALUE associated with the received message. This is the clear case of an error:
since the tuple (M,l) is located in HISTORY, it is inferred that the MESSAGE has been received previously;
the fact that associated VALUES are not the same proves that either the sender had been faulty and had used the same message identifier for two distinct message broadcasts, or the authentication protocol has been compromised. It is therefore determined that MESSAGE is invalid and must not be accepted. In order to achieve this, the following steps are undertaken: (1) the located tùple ~M,1) is modified to (M,2) so as to record the invalidity of the message (havin~ at least two associated authentic ~ALUEs; this will enable the local END task to distinguish the invalid messages and avoid their acceptance); and (2) the received (signed) message X is signed and the result broadcast to all of the neighbors except the one from whom the message X was received. In step (2), processor s signs the MESSAGE and sends it to all of its neighbors except the one from whom the MESSAGE
was received. The latter step is necessary to inform other fault-free processors about the invalidity of the MESSAGE.
(d) A tuple (M,2) is found. This is the case where it has already been determined that a MESSAGE is invalid (see the case above3. Since all necessary actions have already been taken, the received MESSAGE is simply discarded.
3~2 The END task is responsible for accepting valid messages, and removing messages from the HISTORY so as to keep it from growing infinitely large. The END task is normally in a wait mode, and is scheduled by interval timer (set by SEND or RECEIVE tasks). When it is scheduled at some time T, it processes in increasing order of MESSAGE IDENTIFIERs all MESSAGEs that were scheduled for processing at this time, i.e. all MESSAGEs with TIMESTAMP equal to T-BYZT. For each such MESSAGE, the END
task removes the tuple (MESSAGE,NUMBER OF VALUES) from the local HISTORY. If the NU~IBER OF VALUES portion of the tuple i5 1, then the MESSAGE VALUE is valid, i.e. only one authentic value has been received; the END task will accept the MESSAGE. Alternatively, if NUMBER OF VALUES
portion is 2 then the message is invalid, elther the sender had been faulty or the authentication protocol has been compromised; the message is not accepted.
Observe that for any given MESSAGE, the END tasks of all ault-free processors will process that message at the same clock time TIMESTAMP+BYZT. Also observe that all valid MESSACEs with the same TIMESTAMP are accepted in the same order by all fault-free processors.
The above three tasks, as shown in Figures l, 2, 3, and 4, are expressible in a high-level programming language. A representation of these tasks written in a high-level specification language is appended to this specification respectively as counterpart Figures 5, 6, and 7. The inclusion of these executable tasks is set forth in order to illustrate, by way of example, the e~se with which the method of this invention may be practiced by those possessing skill in this art. Other high-level language representations such as PL/I, PASCAL, or ADA
might likewise with equal ease and facility have been drawn.
'3 '~J~3 Illustrative Examples Example 1 This example exhibits the method's operation in the absence of faults in a network of three interconnected processors as depicted in Eigure 8. All three processors are assumed to be fault free.
(a) At some time To on S's clock, S initiates the broadcast of the MESSAGE ::= (ID,To,VALUE), where ID is a unique message identifier. S stores (MESSAGE,1) in its HISTORY, sets the interval timer to schedule its END task at To+BYZT, siyns the MESSAGE, and broadcasts the resulted message X on links 1 and 3.
(b) At some time Tl on P's clock, where To - DMAX < Tl ~ T~ + DMAX ~ TDEL, processor P receives the message X on link 1, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s), s being processor S's signature. This MESSAGE passes all acceptance tests. P
stores the tuple (MESSAGE,l) in its HISTORY, sets its interval timer to schedule its END task at To+BYZT, signs the received messaye X, and broadcasts the resulting message Y on link 3.
(c) At some time T2 on Q's clock, where To DMAX T2 ~o DMAX TDEL, processor Q receives the message X on link 2, verifies its authenticity and derives rom it the original MESSAGE and the seguence of signatures (s). This MESSAGE passes all acceptance tests. Q stores the tuple (MESSAGE,1) in its HISTORY, sets its interval timer to schedule its END task at To+BYZT, signs the message X, and broadcasts the resulting message Z on link 3.
S (d) At some time T3 on P's clock, where To = DMAX < T3 ~ To + D~AX + (2 x TDEL), processor P receives the message Z on link 3, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s,q). This MESSAGE passes all acceptance tests. However, since the tuple (MESSAGE,1) is found in P's HISTORY, P ignores the second receipt of MESSAGE.
te) At some time T4 on Q's clock, where To - DMAX ~ T4 ~ To + DMAX + (2 x TDEL), processor Q receives the message Y on link 3, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s,p). This MESSAGE.passes all acceptance tests. However, since the tuple (MESSAGE,l) is found i.n Q's HISTORY, Q ignores the second receipt of MESSAGE.
~ f) At time To+BYZT on S, P, and Q's clocks, the END
tasks of all three processors are scheduled. The MESSAGE
VALUE is accepted by all three processors, and the tuple (MESSAGE,1) is removed from all three HISTORYs.
Example 2 This example exhibits the method's operation in the presence of faults in a network of three interconnected processors as depicted in Figure 8. In this example, S processors S and P are assumed to be fault free, while processor Q is assumed to be experiencing "timing"
faults.
(a) At some time To on S's cloclc, S initiates the broadcast of the MESSAGE ::= (ID,To,VALUE). S stores (MESSAGE,l) in its HISTORY, sets the interval timer to schedule its ~ND task at To+BYZT, signs the MESSAGE, and broadcasts the resulting message X on links 1 and 3.
(b) At some time Tl on P's clock, where To - DMAX ~ Tl ~ To + DMAX + TDEL~
processor P receives the message X on link 1, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s). This MESSAGE passes all acceptance tests. P stores the tuple (MESSAGE~l) in its HISTORY, sets its interval timer to schedule its END task at To+BYZT, signs the message X, and broadcasts the resulting message Y on link 3.
(c) At some time T2 on Q's clock, where To - DMAX < T2 ~ To + DMAX ~ TDEL, processor Q receives the message X on link 2, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s). This MESSAGE passes all acceptance tests. Q stores the tuple (MESSAGE,l) in its HISTORY, and sets its interval timer to schedule its END
task at To+BYZT.
Suppose that at this time processor Q is affected by a "timing" fault and becomes inoperable for a long time.
(d) At time To+BYZT on the clock of processors S and P, the END tasks of both processors are scheduled. The MESSAGE VALUE is accepted by both processors, and the tuple (MESSAGE,l) is removed from both HISTORYs.
(e) At some time T3, much later after S and P have accepted the MESSAGE VALUE, Q finally becomes operable, signs the message X, and broadcasts the resulting message Z on link 3.
(f) At some time T4 on P's clock, P receives the message Z on lin~ 3, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s,~). Since P has already accepted this MESSAGE VALUE, it follows that T4 > To+BYZT. Therefore, the newly received MESSAGE will be rejected by the test for late messa~e arrival (test T3 of RECEIVE task, see Figures 2 and 3); the MESSAGE will be discarded.
The final result in the above example i5. that the ~0 fault-free processors S and P accept the MESSAGE VALUE
exactly once.
"Timing" Faults If the fault class to be handled by the method of this invention is restricted to "timing" faults, then a simpler method can be derived from the method of this invention. Specifically, considerable reduction in complexity results because (1) message authentication is no longer needed, typically a complex operation; (2) there is no longer a need to protect against multi-~alued messages; and (3) the tests for bad signatures are - ~2~3~
replaced by a much simpler test. The simpler method can be derived by making the following changes to the method of this invention:
(a) The format of a ~IESSAGE is chanyed to include a count of the processors that have received that message, hereafter referred to as the HOP-COUNT.
(b) In the SEND task, the value of the HOP-COUNT is set to 1 by the MESSAGE originator prior to sending it to all neighbors.
10(c) In the RECEIVE task, the value of the HOP-COUNT
in a MESSAGE is incremented by 1 by all intermediate processors which receive that MESSAGE prior to sending it to all neighbors.
(d) The format of the HISTORY tuples is changed to <MESSAGE>. The field NUMBER OF VALUES is no longer needed.
(e) A MESSAGE is not signed by the SEND before its transmission, or by the RECEIVE before relaying it.
(f) Test Tl in the RECEIVE task is no longer needed.
20(g) Test T2 in the REGEIVE task is changed to "check if HOP-COUNT ~ F+DIAM(F)".
.
(h) Test T3 in the RECEIVE task is changed to "check if the difference between the local time and TIMESTAMP is greater than the minimum of HOP-COUNT x (DMAX + TDEL) and BYZT".
(i) Test T4 in the RECEIVE task is changed to "check if the local time is smaller than TIMESTAMP minus HOP-COUNT x DMAX'I. `
33~'~
(j) Test T5 in the RECEIVE task is changed to "Search the local HISTORY for the existence of a tuple of the form CMESSAGE~".
(k~ Tests TG and T7 in the RECEIVE task are no longer needed.
(l) In the END task, the test "NUMBER OF VALUES =
1?" is no longer needed.
For the above derived method, a representation of the SEND task, the RECEIVE task, and the END task written in a high-level language is appended to this specification respectively as counterpart Figures 9, 10, and 11.
"Omission" Faults If the ault class to be handled by the method of this invention is restricted to ~'omission" faults, then a yet simpler method can be derived from the method of this invention. Specifically, considerable reduction in complexity results because (1) message authentication is no longer needed, typically a complex operation; (2~
there is no longer a need to protect against multi-valued messages; (3) tests for bad signatures are no longer needed; and (43 tests for message timeliness are replaced by a single and very simple test. The simpler method can be derived by making the following changes to the method of this invention:
(a) The format of the HISTORY tuples is changed to ~MESSAGE>. The field NUMBER 0~ VALUES is no longer needed.
3~
(b) A MESSAGE is not signed by the SE~ before its transmission, or by the RECEIVE before relaying it.
(c) Tests Tl, T2, T3, T4, T6, and T7 in the RECEIVE
task are no longer needed.
S (d) Test T5 in the RECEIVE task is changed to "Search the local HISTORY for the existence of a tuple of the form <MESSAGE~".
(e) The SEND task schedules the E~ task at time TIMESTAMP~DMAX~DIAM(F)xTDEL, rather than TIMESTAMP+BYZT.
(f) The RECEIVE task schedules the END task at time TIMESTAMP~DMAX~DIAM(F)xTDEL, rather than TIMESTAMP+BYZT.
(g) In the END task, the test "NUMBER OF VALUES =
l?" is no longer needed.
For the above-derived method, a representation of the SEND task, the RECEIVE task, and the END task written in a high-level language is appended to this specification respectively as counterpart Figures 12, 13, and 14.
Furthermore, if the order of message acceptance by different processors is not important, then processor clocks need no longer be (approximately) synchronized.
Removing this, yet another source of complexity, a simpler method can be derived from the method of this invention by maXing the following changes:
(a) Synchronized clocks are no longer needed. Each processor requires an interval timer only.
~ SA9-84-017 -21-~33~
(b) The format of the HISTORY tuples is changed to <MESSAGE>. The field NUMBER OF VALUES is no longer needed.
(c) A MESSAGE is not signed by the SEND before its transmission, or by the RECEIVE beore relaying it.
(d) The SEND task accepts a message as soon as that message broadcast request is received.
(e) The SEND task schedules the END task to execute (DIAM(F)+l)xTDEL time units after receiving a message from a local sender (so that it removes that MESSAGE from local HISTORY).
(f) Tests T1, T2, T3, T4, T6, and T7 in the RECEIVE
task are no longer needed.
(g) Test T5 in the RECEIVE task is changed to "Search the local HISTORY for the existence of a tuple of the form <MESSAGE>".
(h) The RECEIVE task schedules the END ~o execute (DIAM(E)+l)xTDEL time units after receiving a MESSAGE
from the network (so that it removes that MESSAGE from local HISTORY).
(i) The END task is extremely simplified and performs only one operation of removing MESSAGEs from the local HISTORY.
For the above-derived method, a representation of the SEND task, the RECEIVE task, and the END task written in a high-level language is appended to this specification respectively as counterpart Figures 15, 16, and 17.
~337Z
From the description of the Preferred Embodiment of this invention, those skilled in this art will recognlze a variety of applications for the invention and appropriate modificatlons within the scope of the claims.
.
The SEND task on each processor s is responsible for initiating the broadcast of messages. Upon receipt of a ~roadcast request, the SEND task executes the following steps of: (1) generating an IDENTIFIER and a TIMESTAMP
5 for that message, to produce the MESSAGE
(IDENTIFIER,TIMESTAMP,VALUE);
~2) storing the tuple <MESSAGE,1> in its local HISTORY;
(3) setting its local interval timer to schedule the END
task at TIMESTAMP+BYZT to accept the MESSAGE VALUE and removing the MESSAGE from HISTORY (observe that by the time TIMESTAMP+BYZT the message should have been received by all fault-free processors); and finally (4) signing the MESSAGE and broadcasting the result to all of its neighbors.
The RECEIVE task on each processor is constantly waiting to receive messages. The main responsibility of the RECEIVE task is to filter out invalid messages, and store each valid message in local HISTORY the first time it is received along with a count of the number of diferent values that are received with that message IDENTIFIER. Observe that a fault-free sender associates an IDENTIFIER with only one message value, whereas a faulty sender may associate several message values with the same IDENTIFIER. It is only the former kind that must be accepted, while the latter must be discarded.
.
Specifically, upon receipt of a signed message X
from the network by a processor s, the RECEIVE task on that processor verifies its authenticity and derives from it the original MESSAGE and a sequence of signatures of the processors that have signed that message (in the order that they have signed that message with sender's signature being the first signature). If the message is determined to be "forged", it will be discarded. If the message is determined to be "authentic", then the RECEIVE
task will execute the following tests: (1) whether the MESSAGE has duplicate signatures; (2) whether the MESSAGE
has more signatures than F~DIAM(F), i.e. too many hops;
(3) whether the MESSAGE has arrived too early, possibly due to some clock failure; and (4) whether the MESSAGE has arrived too late, possibly due to timing faults. If the answer to any one of the above tests is "yes", then the received message is determined to be invalid and is, therefore, discarded.
Once the message passes all of the above tests, then the RECEIVE task checks if it has already seen this message by consulting its HISTORY. That is, the HISTORY
is searched for a tuple (M,NUMBER OF VALUES) such that M's IDENTIFIER is the same as MESSAGE's IDENTIFIER. The search results in one of the following outcomes:
(a) No such tuple is found; this signifies that this MESSAGE is received for the first time and has not been seen before. The RECEIVE task then executes the following steps of: (1) storing the tuple (ME~SAGE,l) in the HISTORY to record the fact that MESSAGE has been received already once and has only one value associated with it; (2) setting its local interval timer to schedule the END task at time TIMESTAMP+BYZT to process the MESSAGE
(at that time, the END task will remove the MESSAGE from local HISTORY, and accepts the message value there is only one value associated with the MESSAGE); and finally (3) signing the received (signed) message X and broadcasting the result to all o its neighbors except the one from whom the message X was received.
(b) A tuple (M,l) is found such that its associated VALUE is the same as the VALUE associated with the ~3'7~
received MESSAGE. This represents the case where another copy of the same message is received again, and may happen since several copies of a message travel the network over different paths and reach the same processor at different times. Since the received MESS~GE has already been seen, it will be discarded.
(c) A tuple (M,l) is found such that its associated value is not the same as the VALUE associated with the received message. This is the clear case of an error:
since the tuple (M,l) is located in HISTORY, it is inferred that the MESSAGE has been received previously;
the fact that associated VALUES are not the same proves that either the sender had been faulty and had used the same message identifier for two distinct message broadcasts, or the authentication protocol has been compromised. It is therefore determined that MESSAGE is invalid and must not be accepted. In order to achieve this, the following steps are undertaken: (1) the located tùple ~M,1) is modified to (M,2) so as to record the invalidity of the message (havin~ at least two associated authentic ~ALUEs; this will enable the local END task to distinguish the invalid messages and avoid their acceptance); and (2) the received (signed) message X is signed and the result broadcast to all of the neighbors except the one from whom the message X was received. In step (2), processor s signs the MESSAGE and sends it to all of its neighbors except the one from whom the MESSAGE
was received. The latter step is necessary to inform other fault-free processors about the invalidity of the MESSAGE.
(d) A tuple (M,2) is found. This is the case where it has already been determined that a MESSAGE is invalid (see the case above3. Since all necessary actions have already been taken, the received MESSAGE is simply discarded.
3~2 The END task is responsible for accepting valid messages, and removing messages from the HISTORY so as to keep it from growing infinitely large. The END task is normally in a wait mode, and is scheduled by interval timer (set by SEND or RECEIVE tasks). When it is scheduled at some time T, it processes in increasing order of MESSAGE IDENTIFIERs all MESSAGEs that were scheduled for processing at this time, i.e. all MESSAGEs with TIMESTAMP equal to T-BYZT. For each such MESSAGE, the END
task removes the tuple (MESSAGE,NUMBER OF VALUES) from the local HISTORY. If the NU~IBER OF VALUES portion of the tuple i5 1, then the MESSAGE VALUE is valid, i.e. only one authentic value has been received; the END task will accept the MESSAGE. Alternatively, if NUMBER OF VALUES
portion is 2 then the message is invalid, elther the sender had been faulty or the authentication protocol has been compromised; the message is not accepted.
Observe that for any given MESSAGE, the END tasks of all ault-free processors will process that message at the same clock time TIMESTAMP+BYZT. Also observe that all valid MESSACEs with the same TIMESTAMP are accepted in the same order by all fault-free processors.
The above three tasks, as shown in Figures l, 2, 3, and 4, are expressible in a high-level programming language. A representation of these tasks written in a high-level specification language is appended to this specification respectively as counterpart Figures 5, 6, and 7. The inclusion of these executable tasks is set forth in order to illustrate, by way of example, the e~se with which the method of this invention may be practiced by those possessing skill in this art. Other high-level language representations such as PL/I, PASCAL, or ADA
might likewise with equal ease and facility have been drawn.
'3 '~J~3 Illustrative Examples Example 1 This example exhibits the method's operation in the absence of faults in a network of three interconnected processors as depicted in Eigure 8. All three processors are assumed to be fault free.
(a) At some time To on S's clock, S initiates the broadcast of the MESSAGE ::= (ID,To,VALUE), where ID is a unique message identifier. S stores (MESSAGE,1) in its HISTORY, sets the interval timer to schedule its END task at To+BYZT, siyns the MESSAGE, and broadcasts the resulted message X on links 1 and 3.
(b) At some time Tl on P's clock, where To - DMAX < Tl ~ T~ + DMAX ~ TDEL, processor P receives the message X on link 1, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s), s being processor S's signature. This MESSAGE passes all acceptance tests. P
stores the tuple (MESSAGE,l) in its HISTORY, sets its interval timer to schedule its END task at To+BYZT, signs the received messaye X, and broadcasts the resulting message Y on link 3.
(c) At some time T2 on Q's clock, where To DMAX T2 ~o DMAX TDEL, processor Q receives the message X on link 2, verifies its authenticity and derives rom it the original MESSAGE and the seguence of signatures (s). This MESSAGE passes all acceptance tests. Q stores the tuple (MESSAGE,1) in its HISTORY, sets its interval timer to schedule its END task at To+BYZT, signs the message X, and broadcasts the resulting message Z on link 3.
S (d) At some time T3 on P's clock, where To = DMAX < T3 ~ To + D~AX + (2 x TDEL), processor P receives the message Z on link 3, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s,q). This MESSAGE passes all acceptance tests. However, since the tuple (MESSAGE,1) is found in P's HISTORY, P ignores the second receipt of MESSAGE.
te) At some time T4 on Q's clock, where To - DMAX ~ T4 ~ To + DMAX + (2 x TDEL), processor Q receives the message Y on link 3, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s,p). This MESSAGE.passes all acceptance tests. However, since the tuple (MESSAGE,l) is found i.n Q's HISTORY, Q ignores the second receipt of MESSAGE.
~ f) At time To+BYZT on S, P, and Q's clocks, the END
tasks of all three processors are scheduled. The MESSAGE
VALUE is accepted by all three processors, and the tuple (MESSAGE,1) is removed from all three HISTORYs.
Example 2 This example exhibits the method's operation in the presence of faults in a network of three interconnected processors as depicted in Figure 8. In this example, S processors S and P are assumed to be fault free, while processor Q is assumed to be experiencing "timing"
faults.
(a) At some time To on S's cloclc, S initiates the broadcast of the MESSAGE ::= (ID,To,VALUE). S stores (MESSAGE,l) in its HISTORY, sets the interval timer to schedule its ~ND task at To+BYZT, signs the MESSAGE, and broadcasts the resulting message X on links 1 and 3.
(b) At some time Tl on P's clock, where To - DMAX ~ Tl ~ To + DMAX + TDEL~
processor P receives the message X on link 1, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s). This MESSAGE passes all acceptance tests. P stores the tuple (MESSAGE~l) in its HISTORY, sets its interval timer to schedule its END task at To+BYZT, signs the message X, and broadcasts the resulting message Y on link 3.
(c) At some time T2 on Q's clock, where To - DMAX < T2 ~ To + DMAX ~ TDEL, processor Q receives the message X on link 2, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s). This MESSAGE passes all acceptance tests. Q stores the tuple (MESSAGE,l) in its HISTORY, and sets its interval timer to schedule its END
task at To+BYZT.
Suppose that at this time processor Q is affected by a "timing" fault and becomes inoperable for a long time.
(d) At time To+BYZT on the clock of processors S and P, the END tasks of both processors are scheduled. The MESSAGE VALUE is accepted by both processors, and the tuple (MESSAGE,l) is removed from both HISTORYs.
(e) At some time T3, much later after S and P have accepted the MESSAGE VALUE, Q finally becomes operable, signs the message X, and broadcasts the resulting message Z on link 3.
(f) At some time T4 on P's clock, P receives the message Z on lin~ 3, verifies its authenticity and derives from it the original MESSAGE and the sequence of signatures (s,~). Since P has already accepted this MESSAGE VALUE, it follows that T4 > To+BYZT. Therefore, the newly received MESSAGE will be rejected by the test for late messa~e arrival (test T3 of RECEIVE task, see Figures 2 and 3); the MESSAGE will be discarded.
The final result in the above example i5. that the ~0 fault-free processors S and P accept the MESSAGE VALUE
exactly once.
"Timing" Faults If the fault class to be handled by the method of this invention is restricted to "timing" faults, then a simpler method can be derived from the method of this invention. Specifically, considerable reduction in complexity results because (1) message authentication is no longer needed, typically a complex operation; (2) there is no longer a need to protect against multi-~alued messages; and (3) the tests for bad signatures are - ~2~3~
replaced by a much simpler test. The simpler method can be derived by making the following changes to the method of this invention:
(a) The format of a ~IESSAGE is chanyed to include a count of the processors that have received that message, hereafter referred to as the HOP-COUNT.
(b) In the SEND task, the value of the HOP-COUNT is set to 1 by the MESSAGE originator prior to sending it to all neighbors.
10(c) In the RECEIVE task, the value of the HOP-COUNT
in a MESSAGE is incremented by 1 by all intermediate processors which receive that MESSAGE prior to sending it to all neighbors.
(d) The format of the HISTORY tuples is changed to <MESSAGE>. The field NUMBER OF VALUES is no longer needed.
(e) A MESSAGE is not signed by the SEND before its transmission, or by the RECEIVE before relaying it.
(f) Test Tl in the RECEIVE task is no longer needed.
20(g) Test T2 in the REGEIVE task is changed to "check if HOP-COUNT ~ F+DIAM(F)".
.
(h) Test T3 in the RECEIVE task is changed to "check if the difference between the local time and TIMESTAMP is greater than the minimum of HOP-COUNT x (DMAX + TDEL) and BYZT".
(i) Test T4 in the RECEIVE task is changed to "check if the local time is smaller than TIMESTAMP minus HOP-COUNT x DMAX'I. `
33~'~
(j) Test T5 in the RECEIVE task is changed to "Search the local HISTORY for the existence of a tuple of the form CMESSAGE~".
(k~ Tests TG and T7 in the RECEIVE task are no longer needed.
(l) In the END task, the test "NUMBER OF VALUES =
1?" is no longer needed.
For the above derived method, a representation of the SEND task, the RECEIVE task, and the END task written in a high-level language is appended to this specification respectively as counterpart Figures 9, 10, and 11.
"Omission" Faults If the ault class to be handled by the method of this invention is restricted to ~'omission" faults, then a yet simpler method can be derived from the method of this invention. Specifically, considerable reduction in complexity results because (1) message authentication is no longer needed, typically a complex operation; (2~
there is no longer a need to protect against multi-valued messages; (3) tests for bad signatures are no longer needed; and (43 tests for message timeliness are replaced by a single and very simple test. The simpler method can be derived by making the following changes to the method of this invention:
(a) The format of the HISTORY tuples is changed to ~MESSAGE>. The field NUMBER 0~ VALUES is no longer needed.
3~
(b) A MESSAGE is not signed by the SE~ before its transmission, or by the RECEIVE before relaying it.
(c) Tests Tl, T2, T3, T4, T6, and T7 in the RECEIVE
task are no longer needed.
S (d) Test T5 in the RECEIVE task is changed to "Search the local HISTORY for the existence of a tuple of the form <MESSAGE~".
(e) The SEND task schedules the E~ task at time TIMESTAMP~DMAX~DIAM(F)xTDEL, rather than TIMESTAMP+BYZT.
(f) The RECEIVE task schedules the END task at time TIMESTAMP~DMAX~DIAM(F)xTDEL, rather than TIMESTAMP+BYZT.
(g) In the END task, the test "NUMBER OF VALUES =
l?" is no longer needed.
For the above-derived method, a representation of the SEND task, the RECEIVE task, and the END task written in a high-level language is appended to this specification respectively as counterpart Figures 12, 13, and 14.
Furthermore, if the order of message acceptance by different processors is not important, then processor clocks need no longer be (approximately) synchronized.
Removing this, yet another source of complexity, a simpler method can be derived from the method of this invention by maXing the following changes:
(a) Synchronized clocks are no longer needed. Each processor requires an interval timer only.
~ SA9-84-017 -21-~33~
(b) The format of the HISTORY tuples is changed to <MESSAGE>. The field NUMBER OF VALUES is no longer needed.
(c) A MESSAGE is not signed by the SEND before its transmission, or by the RECEIVE beore relaying it.
(d) The SEND task accepts a message as soon as that message broadcast request is received.
(e) The SEND task schedules the END task to execute (DIAM(F)+l)xTDEL time units after receiving a message from a local sender (so that it removes that MESSAGE from local HISTORY).
(f) Tests T1, T2, T3, T4, T6, and T7 in the RECEIVE
task are no longer needed.
(g) Test T5 in the RECEIVE task is changed to "Search the local HISTORY for the existence of a tuple of the form <MESSAGE>".
(h) The RECEIVE task schedules the END ~o execute (DIAM(E)+l)xTDEL time units after receiving a MESSAGE
from the network (so that it removes that MESSAGE from local HISTORY).
(i) The END task is extremely simplified and performs only one operation of removing MESSAGEs from the local HISTORY.
For the above-derived method, a representation of the SEND task, the RECEIVE task, and the END task written in a high-level language is appended to this specification respectively as counterpart Figures 15, 16, and 17.
~337Z
From the description of the Preferred Embodiment of this invention, those skilled in this art will recognlze a variety of applications for the invention and appropriate modificatlons within the scope of the claims.
.
Claims (13)
1. A method for reliably broadcasting information in a point-to-point network of message exchanging processors, each processor having an amendable log, each processor having a clock approximately synchronized to the clocks of the other processors, the time taken for the transmission of messages between arbitrary points in the network and processing thereof being bounded, comprising the steps of:
(a) responsive to a message broadcast request originating at any processor s, formatting a message M and assigning thereto a network unique identifier and a timestamp; adding message M to the processor s's log; signing message M including the assigned values to form message X; and broadcasting message X
to network adjacent processors of processor s;
(b) responsive to the receipt of a signed message X by each processor p from a network adjacent processor s, verifying its authenticity and deriving from it the original message M and a sequence of signatures of all previous processors that have signed X in the order that they have signed X with the sender's signature being the first signature;
and ascertaining the validity of message M; and (c) at each processor q within the set of processors network adjacent to processor s which finds the message M valid, performing the further steps of appending message M to its log; signing the message X; broadcasting said signed message X to its network adjacent neighbors except the processor from whom the message X was received; and, after a predetermined amount of time from the time of message origination, accepting the message M if it is determined to be valid.
(a) responsive to a message broadcast request originating at any processor s, formatting a message M and assigning thereto a network unique identifier and a timestamp; adding message M to the processor s's log; signing message M including the assigned values to form message X; and broadcasting message X
to network adjacent processors of processor s;
(b) responsive to the receipt of a signed message X by each processor p from a network adjacent processor s, verifying its authenticity and deriving from it the original message M and a sequence of signatures of all previous processors that have signed X in the order that they have signed X with the sender's signature being the first signature;
and ascertaining the validity of message M; and (c) at each processor q within the set of processors network adjacent to processor s which finds the message M valid, performing the further steps of appending message M to its log; signing the message X; broadcasting said signed message X to its network adjacent neighbors except the processor from whom the message X was received; and, after a predetermined amount of time from the time of message origination, accepting the message M if it is determined to be valid.
2. A method according to claim 1, wherein the step of ascertaining message validity includes the steps of determining whether the message is authentic, whether it has not already been signed by processor p, whether it contains duplicate signatures, whether it is nonduplicative of a previously received message, and whether it is timely.
3. A method according to claim 1, in which every message broadcast is either accepted by all fault-free processors or none of them.
4. A method according to claim 1, in which all message broadcasts accepted by fault-free processors are accepted in the same order by all of them.
5. A method for reliably broadcasting information in a point-to-point network of message exchanging processors, each processor having an amendable log, each processor clock being approximately synchronized with the clocks of the other processors, the time for the network transmission of messages and processing being bounded, comprising the steps of:
(a) responsive to a message broadcast request local to any processor s, formatting a message M and assigning to said message a network unique identifier, a timestamp, and a counter initialized to l; adding the message to its local log; and broadcasting message M including the assigned values to the network adjacent processors of processor s;
(b) responsive to receipt of a message by each processor p from a network adjacent processor s, ascertaining message validity as to its being nonduplicative of a previously received message and as to its timeliness; and (c) at each processor q within the set of network adjacent processors which finds the message M valid, performing the further steps of:
(1) appending message M to its log;
(2) incrementing the counter by 1 and broadcasting the message to the network adjacent neighbor processors except the processor from whom the message was received;
and (3) after a predetermined amount of time from the time of message origination, accepting the message M and erasing it from the local log.
(a) responsive to a message broadcast request local to any processor s, formatting a message M and assigning to said message a network unique identifier, a timestamp, and a counter initialized to l; adding the message to its local log; and broadcasting message M including the assigned values to the network adjacent processors of processor s;
(b) responsive to receipt of a message by each processor p from a network adjacent processor s, ascertaining message validity as to its being nonduplicative of a previously received message and as to its timeliness; and (c) at each processor q within the set of network adjacent processors which finds the message M valid, performing the further steps of:
(1) appending message M to its log;
(2) incrementing the counter by 1 and broadcasting the message to the network adjacent neighbor processors except the processor from whom the message was received;
and (3) after a predetermined amount of time from the time of message origination, accepting the message M and erasing it from the local log.
6. A method according to claim 5, in which every message broadcast is either accepted by all fault-free processors or none of them.
7. A method according to claim 5, in which all message broadcasts accepted by fault-free processors are accepted in the same order by all of them.
8. A method for reliably broadcasting information in a point-to-point network of message exchanging processors, each processor having an amendable log, each processor clock being approximately synchronized with the clocks of the other processors, the time of network transmission and processing of messages being bounded, comprising the steps of:
(a) responsive to a message broadcast request local to any processor s, assigning a network unique identifier and a timestamp; adding the message to its local log; and broadcasting that message including the assigned values to network adjacent processors to processor s;
(b) responsive to the reception of a broadcast message by each processor p from a network adjacent processor s, ascertaining message validity as to its being nonduplicative of a previously received message; and (c) at each processor q within the set of processors p which finds the message valid, performing the further steps of:
(1) appending the message to its log;
(2) broadcasting the message to its network adjacent processors except the processor from whom the message was received;
and (3) after a predetermined amount of time from the time of message origination, accepting the message and erasing it from its log.
(a) responsive to a message broadcast request local to any processor s, assigning a network unique identifier and a timestamp; adding the message to its local log; and broadcasting that message including the assigned values to network adjacent processors to processor s;
(b) responsive to the reception of a broadcast message by each processor p from a network adjacent processor s, ascertaining message validity as to its being nonduplicative of a previously received message; and (c) at each processor q within the set of processors p which finds the message valid, performing the further steps of:
(1) appending the message to its log;
(2) broadcasting the message to its network adjacent processors except the processor from whom the message was received;
and (3) after a predetermined amount of time from the time of message origination, accepting the message and erasing it from its log.
9. A method according to claim 8, in which every message broadcast is either accepted by all fault-free processors or none of them.
10. A method according to claim 8, in which all message broadcasts accepted by fault-free processors are accepted in the same order by all of them.
11. A method for reliably broadcasting information in a point-to-point network of message exchanging processors, each processor having an amendable log, the time for message network transmission and processing being bounded, comprising the steps of:
(a) responsive to a message broadcast request local to any processor s, formatting and assigning to a message a network unique identifier; accepting the message; appending the message to its log; and broadcasting the message including the assigned identifier to the processor network adjacent thereto;
(b) responsive to the receipt of a broadcast message at each processor p from a network adjacent processor s, ascertaining message validity as to its being nonduplicative of a previously received message; and (c) at each processor q within the set of processors p which finds the message valid, performing the further steps of:
(1) accepting the message;
(2) updating its message log by appending the received message thereto;
(3) broadcasting the message to its network adjacent processors except the processor from whom the message was received;
and (4) after a predetermined amount of time from the time of message origination, erasing that message from the log.
(a) responsive to a message broadcast request local to any processor s, formatting and assigning to a message a network unique identifier; accepting the message; appending the message to its log; and broadcasting the message including the assigned identifier to the processor network adjacent thereto;
(b) responsive to the receipt of a broadcast message at each processor p from a network adjacent processor s, ascertaining message validity as to its being nonduplicative of a previously received message; and (c) at each processor q within the set of processors p which finds the message valid, performing the further steps of:
(1) accepting the message;
(2) updating its message log by appending the received message thereto;
(3) broadcasting the message to its network adjacent processors except the processor from whom the message was received;
and (4) after a predetermined amount of time from the time of message origination, erasing that message from the log.
12. A method according to claim 11, wherein different processors may accept messages in different order.
13. A method according to claim 11, wherein every message broadcast is either accepted by all fault-free processors or none of them.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US06/661,514 US4644542A (en) | 1984-10-16 | 1984-10-16 | Fault-tolerant atomic broadcast methods |
| US661,514 | 1984-10-16 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1223372A true CA1223372A (en) | 1987-06-23 |
Family
ID=24653915
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000482185A Expired CA1223372A (en) | 1984-10-16 | 1985-05-23 | Fault-tolerant atomic broadcast methods |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US4644542A (en) |
| EP (1) | EP0178473B1 (en) |
| JP (1) | JPS6198037A (en) |
| CA (1) | CA1223372A (en) |
| DE (1) | DE3584816D1 (en) |
Families Citing this family (37)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4718002A (en) * | 1985-06-05 | 1988-01-05 | Tandem Computers Incorporated | Method for multiprocessor communications |
| US4766534A (en) * | 1986-10-16 | 1988-08-23 | American Telephone And Telegraph Company, At&T Bell Laboratories | Parallel processing network and method |
| GB2203617B (en) * | 1987-03-30 | 1991-08-21 | Ind Technology Inst | Embedded test system for communications systems conformance testing |
| US4807224A (en) * | 1987-08-21 | 1989-02-21 | Naron Steven E | Multicast data distribution system and method |
| EP0325384B1 (en) * | 1988-01-15 | 1993-09-29 | Quantel Limited | Data processing and communication |
| US5109384A (en) * | 1988-11-02 | 1992-04-28 | Tseung Lawrence C N | Guaranteed reliable broadcast network |
| US5377322A (en) * | 1989-07-19 | 1994-12-27 | Hitachi, Ltd. | Information handling method and system utilizing multiple interconnected processors and controllers |
| JPH03130842A (en) * | 1989-10-17 | 1991-06-04 | Toshiba Corp | Simultaneous execution controller for data base system |
| JPH05167585A (en) * | 1991-12-16 | 1993-07-02 | Nec Corp | Local area network multi-address frame communication device and method |
| JP2534430B2 (en) * | 1992-04-15 | 1996-09-18 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Methods for achieving match of computer system output with fault tolerance |
| US6104871A (en) * | 1996-04-30 | 2000-08-15 | International Business Machines Corporation | Utilizing batch requests to present membership changes to process groups |
| US5704032A (en) * | 1996-04-30 | 1997-12-30 | International Business Machines Corporation | Method for group leader recovery in a distributed computing environment |
| US5787250A (en) * | 1996-04-30 | 1998-07-28 | International Business Machines Corporation | Program product for managing membership of a group of processors in a distributed computing environment |
| US5696896A (en) * | 1996-04-30 | 1997-12-09 | International Business Machines Corporation | Program product for group leader recovery in a distributed computing environment |
| US5787249A (en) * | 1996-04-30 | 1998-07-28 | International Business Machines Coporation | Method for managing membership of a group of processors in a distributed computing environment |
| US5699501A (en) * | 1996-04-30 | 1997-12-16 | International Business Machines Corporation | System for group leader recovery in a distributed computing environment |
| US6052712A (en) * | 1996-04-30 | 2000-04-18 | International Business Machines Corporation | System for barrier synchronization wherein members dynamic voting controls the number of synchronization phases of protocols and progression to each subsequent phase |
| US6016505A (en) * | 1996-04-30 | 2000-01-18 | International Business Machines Corporation | Program product to effect barrier synchronization in a distributed computing environment |
| US6216150B1 (en) | 1996-04-30 | 2001-04-10 | International Business Machines Corporation | Program product for an application programming interface unifying multiple mechanisms |
| US5768538A (en) * | 1996-04-30 | 1998-06-16 | International Business Machines Corporation | Barrier synchronization method wherein members dynamic voting controls the number of synchronization phases of protocols and progression to each new phase |
| US5790772A (en) * | 1996-04-30 | 1998-08-04 | International Business Machines Corporation | Communications method involving groups of processors of a distributed computing environment |
| US5799146A (en) * | 1996-04-30 | 1998-08-25 | International Business Machines Corporation | Communications system involving groups of processors of a distributed computing environment |
| US5793962A (en) * | 1996-04-30 | 1998-08-11 | International Business Machines Corporation | System for managing membership of a group of processors in a distributed computing environment |
| US6026426A (en) * | 1996-04-30 | 2000-02-15 | International Business Machines Corporation | Application programming interface unifying multiple mechanisms |
| US5764875A (en) * | 1996-04-30 | 1998-06-09 | International Business Machines Corporation | Communications program product involving groups of processors of a distributed computing environment |
| US5748958A (en) * | 1996-04-30 | 1998-05-05 | International Business Machines Corporation | System for utilizing batch requests to present membership changes to process groups |
| US7162512B1 (en) * | 2000-02-28 | 2007-01-09 | Microsoft Corporation | Guaranteed exactly once delivery of messages |
| US7171613B1 (en) | 2000-10-30 | 2007-01-30 | International Business Machines Corporation | Web-based application for inbound message synchronization |
| US6931431B2 (en) * | 2001-01-13 | 2005-08-16 | International Business Machines Corporation | Agreement and atomic broadcast in asynchronous networks |
| US7062490B2 (en) * | 2001-03-26 | 2006-06-13 | Microsoft Corporation | Serverless distributed file system |
| GB0123453D0 (en) * | 2001-09-28 | 2001-11-21 | Ncipher Corp Ltd | Time stamping device |
| US20080222234A1 (en) * | 2002-05-23 | 2008-09-11 | Benoit Marchand | Deployment and Scaling of Virtual Environments |
| US7305585B2 (en) * | 2002-05-23 | 2007-12-04 | Exludus Technologies Inc. | Asynchronous and autonomous data replication |
| US20050216910A1 (en) * | 2002-05-23 | 2005-09-29 | Benoit Marchand | Increasing fault-tolerance and minimizing network bandwidth requirements in software installation modules |
| US20050060608A1 (en) * | 2002-05-23 | 2005-03-17 | Benoit Marchand | Maximizing processor utilization and minimizing network bandwidth requirements in throughput compute clusters |
| US7649889B2 (en) * | 2002-07-23 | 2010-01-19 | Vectormax Corporation | Server arbitrated reliable multicast system and process for accessing the same |
| DE10256005A1 (en) * | 2002-08-01 | 2004-02-12 | Daimlerchrysler Ag | Electronic message transmission method for carrying out a messaging process between a sender and a receiver uses temporarily and globally unambiguous identification keys for events and periods |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4276594A (en) * | 1978-01-27 | 1981-06-30 | Gould Inc. Modicon Division | Digital computer with multi-processor capability utilizing intelligent composite memory and input/output modules and method for performing the same |
| US4325120A (en) * | 1978-12-21 | 1982-04-13 | Intel Corporation | Data processing system |
| US4354225A (en) * | 1979-10-11 | 1982-10-12 | Nanodata Computer Corporation | Intelligent main store for data processing systems |
| US4418384A (en) * | 1980-10-06 | 1983-11-29 | Honeywell Information Systems Inc. | Communication subsystem with an automatic abort transmission upon transmit underrun |
| EP0135499B1 (en) * | 1983-02-09 | 1990-05-02 | International Business Machines Corporation | A method for achieving multiple processor agreement optimized for no faults |
-
1984
- 1984-10-16 US US06/661,514 patent/US4644542A/en not_active Expired - Lifetime
-
1985
- 1985-05-23 CA CA000482185A patent/CA1223372A/en not_active Expired
- 1985-08-14 JP JP60177794A patent/JPS6198037A/en active Granted
- 1985-09-17 DE DE8585111750T patent/DE3584816D1/en not_active Expired - Fee Related
- 1985-09-17 EP EP85111750A patent/EP0178473B1/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| US4644542A (en) | 1987-02-17 |
| JPH0377698B2 (en) | 1991-12-11 |
| EP0178473A3 (en) | 1988-11-09 |
| EP0178473B1 (en) | 1991-12-04 |
| EP0178473A2 (en) | 1986-04-23 |
| JPS6198037A (en) | 1986-05-16 |
| DE3584816D1 (en) | 1992-01-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1223372A (en) | Fault-tolerant atomic broadcast methods | |
| Gupta et al. | Resilientdb: Global scale resilient blockchain fabric | |
| Cristian et al. | Atomic broadcast: From simple message diffusion to Byzantine agreement | |
| Hadzilacos et al. | A modular approach to fault-tolerant broadcasts and related problems | |
| CA2205725C (en) | Preventing conflicts in distributed systems | |
| EP0135499B1 (en) | A method for achieving multiple processor agreement optimized for no faults | |
| Ellis | A Robust Algorithm for Updating Duplicate Databases. | |
| Babaoglu et al. | Understanding non-blocking atomic commitment | |
| Chandramouli et al. | A survey on perfectly secure verifiable secret-sharing | |
| Savarí et al. | Finite termination of asynchronous iterative algorithms | |
| Vizier et al. | ComChain: A blockchain with Byzantine fault‐tolerant reconfiguration | |
| Rodrigues et al. | Atomic broadcast in asynchronous crash-recovery distributed systems | |
| Yap et al. | Fault tolerant remote procedure call | |
| Gurevich et al. | Group membership protocol: specification and verification. | |
| Martini et al. | Tolerant synchronization for distributed simulations of interconnected computer networks | |
| Doudou et al. | Tolerating arbitrary failures with state machine replication | |
| Chang et al. | Affordable fault tolerance through adaptation | |
| Misra et al. | Causal ordering properties of byzantine reliable broadcast primitives | |
| Ancilotti et al. | A distributed commit protocol for a multicomputer system | |
| CN112162866A (en) | Heterogeneous executive program synchronization method and device based on relative time | |
| Babaoglu et al. | Reliable broadcasts and communication models: tradeoffs and lower bounds | |
| Ramírez et al. | Distributed control of updates in multiple copy databases: A time optimal algorithm | |
| Hellings et al. | Byzantine Cluster-Sending in Expected Constant Communication | |
| Ramarao | Transaction atomicity in the presence of network partitions | |
| Hellings et al. | On the Correctness of Speculative Consensus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKEX | Expiry |