Pwl rewal-slideshare

Ineluctable modality of the
distributed
On Joseph Halpern’s work on
knowledge in distributed
systems
Peter Alvaro
UC Berkeley

choose-your-own-adventure talk

Last time at PWL…
•  The agreement problem(s)
•  Impossibility results
•  A “weakest” failure detector
Today: knowledge

It’s not just for byzantine stuﬀ
I'm not a great fool, so I can clearly not choose the wine in
front of you. But you must have known I was not a great
fool; you would have counted on it, so I can clearly not
choose the wine in front of me.

Why you should care
A correct distributed program achieves
(nontrivial) distributed property X.
Some tricky questions before we start coding:
1.  Is X even attainable?
2.  Cheapest protocol that gets me X?
3.  How should I implement it?

A strong claim about distributed
correctness properties

Uncertainty is what makes reasoning about
distributed systems diﬃcult.
Uncertainty is the abundance of possibilities.
Knowledge is the dual of possibility

A strong statement about
protocols
How: Protocols just describe what actions to
take based on local knowledge.
Why: Protocols are just mechanisms to
ensure that a group has shared knowledge of
a fact.

A good paper about bridging the gap
between properties and protocols

For example
•  Commit protocols
– each agent knows the commit/abort
decision AND knows that all agents know
the decision
•  Distributed garbage collection
– an agent knows that no remote references
exist to a particular object, and that all other
agents know

For example
•  When the leader has received phase 2b messages for
value v and ballot bal from a majority of the acceptors, it
knows that the value v has been chosen. [paxos]
•  a process takes a checkpoint when it knows that all
processes on which it computationally depends took their
checkpoints [An Eﬃcient Protocol for Checkpointing
Recovery in Distributed Systems, Kim and Park]
•  and therefore a cohort with a later viewstamp for some
view knows everything known to a cohort with an earlier
viewstamp for that view. [viewstamped replication]
•  Since each member of Si serves as an arbitrator, the
requesting node knows that it is the only node that has
been granted mutual exclusion [A sqrt(N) Algorithm for
Mutual Exclusion in Decentralized Systems, Maekawa]

Warmup: RPC protocols
Hi!
Alice Bob

Hi!
Alice Bob
Issue: uncertainty!
Uncertain environment è Uncertain outcomes

Alice Bob
Issue: uncertainty!
Uncertain environment è Uncertain outcomes

Hi!
Retry

Alice Bob

Hi!
Issues: inﬁnite (sender) behavior & state,
at-least-once delivery
Retry

Alice Bob

Hi!
Retry with ACKS
Hi!
Alice Bob

Hi!
Retry with ACKS
Hi!Hi!
Alice Bob
Hi!

Hi!
Hi yourself
Retry with ACKS
Hi!
Issues: at-least once delivery
Hi!
Alice Bob
Hi!

Hi!
Hi yourself
Retry with ACKS
Hi!
Hi!
Alice Bob

Retry with ACKS
Alice Bob
Hi!

a
good
paper
about
principled
distributed
GC

Hi!
Issues: inﬁnite receiver state
Receiver buﬀers, dedups
Alice Bob

Issues: inﬁnite receiver state
Hi!
Receiver buﬀers, dedups
Alice Bob

Hi!
ACK-ACKing
Hi!
Alice Bob

Hi!
Hi yourself
ACK-ACKing
Hi!
Issue: uncertainty
Alice Bob

Hi!
Hi yourself
ACK-ACKing
Hi!
Alice Bob

ACK-ACKing
Hi!
Alice Bob
Ahoy

ACK-ACKing
Alice Bob

ACK-ACKing
Issue: uncertainty
Alice Bob

Issues: inﬁnite hot potato
Alice Bob

what does this remind me of?
Refresher: the two generals problem

(propositional) logic
ϕ ϕ if ϕ is atomic
ϕ ∧ ψ true if both ϕ and ψ are true
¬ϕ true if ϕ is false
Sweet duality:
ϕ ∨ ψ = ¬(¬ϕ ∧ ¬ψ)
ϕ ⇒ ψ= ¬(ϕ ∧ ¬ψ)
q ⇒ p
p = “the write is stable”
q = “the write is acknowledged”

modality, duality
∃xϕ === ¬∀x ¬ϕ
¯ϕ === ¬£¬ϕ
Symbol
Temporal
Deon/c
Epistemic

¯
Some8mes
Is
permi:ed
Is
possible

£
Always
Is
obligatory
Is
known

Knowledge is the dual of possibility

Epistemic modal logic
ϕ = “the write is stable”
Kaliceϕ = “alice knows ϕ”
KaliceKbobϕ = “alice knows bob knows ϕ”
KaliceKbobKcarolϕ = “alice knows bob knows
carol knows ϕ”
[…]

Epistemic modal logic
Eϕ = “everyone* knows ϕ”
EEϕ = “everyone knows everyone knows ϕ”
[…]
A driver will not feel safe going when he sees a
green light unless he knows that everyone else
knows and follows the rules.

Common knowledge
Eϕ = “everyone* knows ϕ”
EEϕ = “everyone knows everyone knows ϕ”
[…]
Eiϕ = “(everyone knows * i) ϕ”
Cϕ = E∞ϕ = “it is common knowledge that ϕ”

Distributed knowledge
Dϕ = “ϕ is implicitly known by the group”
Sϕ = “someone knows ϕ”

Protocols
climb
the
hierarchy

Cϕ
[…]
Ek+1ϕ
[…]
Eϕ
Sϕ
Dϕ
ϕ

Protocols
climb
the
hierarchy

Cϕ
[…]
Ek+1ϕ
[…]
Eϕ
Sϕ
Dϕ
ϕ

Deadlock detection
ϕ is distributed knowledge

Someone knows ϕ

Protocols
climb
the
hierarchy

Cϕ
[…]
Ek+1ϕ
[…]
Eϕ
Sϕ
Dϕ
ϕ

Reliable broadcast
Someone knows ϕ

Everyone knows ϕ

Protocols
climb
the
hierarchy

Cϕ
[…]
E3ϕ
E2ϕ
Eϕ
Sϕ
Dϕ
ϕ

Uniform
Reliable broadcast
Someone knows ϕ

Everyone knows ϕ
Everyone knows
everyone knows ϕ

Protocols
climb
the
hierarchy

Cϕ
[…]
E3ϕ
E2ϕ
Eϕ
Sϕ
Dϕ
ϕ

Someone knows ϕ

Everyone knows ϕ
Everyone knows
everyone knows ϕ
Some crazy BFT
protocol
(Everyone knows)k ϕ

Protocols
climb
the
hierarchy

Cϕ
[…]
E3ϕ
E2ϕ
Eϕ
Sϕ
Dϕ
ϕ

Knowledge
Highway

E10ϕ

10

E100ϕ

100

Cϕ
∞

Applications of knowledge
A correct distributed program achieves
(nontrivial) distributed property X.
Some tricky questions before we start coding:
1.  Is X even attainable?
2.  Cheapest protocol that gets me X?
3.  How should I implement it?

Applications: impossibility
“in a system in which communication is not
guaranteed, common knowledge of
initially-undetermined facts is not
attainable in any run of any protocol.”
Corollary: the 2 generals problem is
unsolvable

Let’s use knowledge to prove it!
But ﬁrst… lots of formalism to get through L

Road map for the proof:
1.  Semantics of modal logic
2.  Distributed system model
3.  A quick and easy lemma
4.  Big theorem: Common knowledge is not
attainable via protocol
5.  Lemma 2: if the generals attack, they have
common knowledge of the attack.
6.  Corollary: 2 generals is unsolvable

Semantics: structures
Formulae are well-formed, meaningless
strings of symbols
Structures give meaning to formulae
(in the very narrow sense of making them all either true or false)
S |= ϕ

Semantics:
propositional structures
Propositional formula:
S |= p ∧ q
Need:
1.  a map S from variable names to T/F
2.  rules; e.g. S |= ϕ ∧ ψ iﬀ S |= ϕ and S |= ψ

Semantics:
ﬁrst-order structures
First-order formula:
S |= ∀x, dog(x) ⇒ big(x) ∧ likes(x, me)
Need:
1.  S assigns “records” to dog, big and likes.
2.  Rules; e.g. S |= ∀xφ iﬀ for all d ∈
|S|,
S[x
:=
d]
|=
φ

Semantics:
ﬁrst-order structures
•  First-order logic:
S |= ∀x, dog(x) ⇒ big(x) ∧ likes(x, me)
dog

Rex

Fido

Rover

big

Rex

Fido

me

likes

Rex
me

Fido
me

Rover
me

me
me

couple good papers about using FO
logic to program distributed systems

Semantics – modal logic
S |= (£¬p) ∧ (q ⇒ ¯r)
Need: a structure that can interpret the
propositional formulae under diﬀerent modalities
Kripke structure: (W, π, R)
•  W is a set of worlds
•  For each element of W, π is a propositional structure
•  R is an accessibility relation among elements of W
S1
S3

Temporal logic
S |= (£¬p) ∧ (q ⇒ ¯r)

q

r

r
q

S1
S3

S2


Epistemic logic
S |= r ∧ ¬Kir ∧ Ki(Kjr or Kj¬r) ∧ Kjr ∧ ¬Kj¬Kir

q

r

r
q

S1
S3

S2

i
j

Kripke structure: (W, π, Ri)

a model of distributed systems
(r,t)
p1 p2 p3 p4 Idealized time
}h(p4,r,t)
A run
r ∈ R

Knowledge-based interpretations
Knowledge interpretation: I = (R, π, {v1,v2,[..]})
Knowledge point: (I, r, t)
R – a set of runs
π – assigns a truth assignment to propositions
for each point in R
vi – A view function for R for some agent i
(determined by h)

choose-your-own-adventure
•  If you’d like to gloss over
the proof and skip to other
applications of knowledge,
turn to page 62
•  If you’d like to dive into the
weeds, turn to page 54.

Truth in a knowledge interpretation
(I,r,t) |= Cφ iﬀ (I,r’,t’) |= Ekφ for all k
Fixed point axiom:
Cφ = E(φ ∧ Cφ)
Induction rule:
From φ ⇒ E(φ ∧ ψ) infer φ ⇒ Cψ

communication is not guaranteed

NG1: For all runs r and times t, there exists a
run r’ extending (r,t) such that […] no messages
are received in r’ at or after time t.
NG2: If in run r processor pi does not receive
any messages in the interval (t’,t), then there is
a run r’ extending (r,t’) such that […] h(pi,r,t’’) =
h(pi,r’,t’’) for all t’’ < t, and no processor pj != pi
receives a message in r’ in the interval (t’,t).

Lemma 1
If, in two diﬀerent runs (r and r’) of the same
protocol, some h(p, r, t) = h(p, r’, t), then
(I, r, t) |= Cφ iﬀ (I, r’, t) |= Cφ
Sorry, no proof today!

Common knowledge is not attainable in a system in
which communication is not guaranteed
Take runs r and r- in R, with the same initial
configuration, s.t. no messages are received in r-
up till time t. Then (I,r,t) |= Cφ iff (I,r-,t) |= Cφ.
Proof (by induction on d(r)*):

•  Base case: d(r)=0. h(p1,r,t) = h(p1,r-,t). By Lemma
1, (I,r,t) |= Cφ iff (I,r-,t) |= Cφ.
*
d(r)
is
the
number
of
messages
received
in
run
r.

Inductive case: d(r) = k+1. Let:

•  t’ < t -- the latest time a message is received in r before t.
•  pj -- a processor that received a message at t’
•  pi –a processor (!= pj)
By NG2, there is a run r’ extending (r,t’) s.t. h(pi,r,t’’)=h(pi,r’,t’’) for
all t’’ <= t, and all processors (besides pi) receive no messages
in the interval (t’, t).
By construction, d(r’) <= k, so by the IH (I,r’,t) |= Cφ iff (I,r-,t) |= Cφ.
But since h(pi,r,t) = h(pi,r’,t), by Lemma 1 (I,r’,t) |= Cφ iff (I,r,t) |= Cφ.
So (I,r,t) |= Cφ iff (I,r-,t) |= Cφ.
QED

Review: we showed that common knowledge cannot be
gained (or lost) by exchanging messages.
Corollary: the 2 generals will never attack.
But we still need to prove one more lemma:
Any correct protocol for coordinated attack has the
property that whenever the generals attack, it is common
knowledge that they are attacking.

Lemma 2: coordinated attack
requires common knowledge
Let ψ = the generals are attacking
Assume the generals (A and B) attack at (r*, t*) – we show that
(I,r*,t*) |= Cψ.
Pick an arbitrary point (r,t). We show ψ ⇒ Eψ is valid in R.
•  If (I,r,t) |= ψ, then the generals attack at (r,t). Consider (r’,t’), in
which A has the same history at (r,t). Since the protocol is
deterministic (assumption), A must also attack in (r’,t’); since
the protocol is correct, B does also, and so (I,r’,t’) |= ψ. It
follows that (I,r,t) |= Eψ, so ψ ⇒ Eψ is valid in R.
•  If (I,r,t) |= ¬ψ, then trivially ψ ⇒ Eψ is valid in R.
By the induction rule, ψ ⇒ Cψ is valid in R

Coup de grace
ψ = the generals are attacking
1.  By assumption, Cψ does not hold if no
messages are exchanged.
2.  By theorem 1, Cψ will never hold.
3.  By lemma 2, the generals cannot attack
unless Cψ.

Phew. but…?
Common knowledge is a prerequisite for agreement.
Common knowledge is not attainable via protocol.

Halpern:
These results may seem paradoxical.

Reality check
Fragile assumptions on which the proofs rest:
•  Deterministic protocol
•  Simultaneous agreement is necessary
•  “Communication not guaranteed”
•  Lack of useful a priori common knowledge

Bootstrapping common knowledge
•  The ``weakest failure detector’’
•  Spanner’s global clock
•  Sequence wraparound

lower bounds for protocols
[Hadzilacos, PODS’87]: A knowledge-theoretic
analysis of atomic commitment protocols
1.  All of the variants of 2pc ((de-)centralized,
linear/nested, etc) are identical from a
knowledge perspective
2.  All 2PC variants attain the minimum level of
knowledge needed to commit
3.  3PC attains the minimum needed to commit
without blocking
4.  Lower bound for messages: nested 2PC.

A good paper about automatically
choosing cheap coordination mechanisms

protocol implementation / synthesis
•  Halpern and Fagin: knowledge-based programming
[PODC’95]

case
of

K(Msg)
and
(KE(AckedMsg))
do
deliver(Msg)

K(Msg)
and
!KE(AckedMsg)
do
relay(Msg)

end

•  Matteo interlandi [Datalog2.0’11]:
Knowlog: knowledge-enriched Dedalus

log(Tx_id,"abort")@next
:-‐
Dvote(Vote,Tx_id),Vote=="no",

par8cipants(X),transac8on(Tx_id,State),State=="vote-‐req".

Monotonicity and knowledge
Monotonic:
the more you know,
the more you know.
Cϕ
[…]
E3ϕ
E2ϕ
Eϕ
Sϕ
Dϕ
ϕ

A good paper about monotonicity
and distributed consistency

Remember
•  Knowledge is the dual of possibility
•  Local knowledge dictates protocol
behavior
•  The purpose of protocols is obtaining a
particular level of distributed knowledge
•  Deep connections between semantic
structures and system behavior
•  Common knowledge is unattainable via
protocol (but there is still hope)

Pwl rewal-slideshare

Recommandé

Recommandé

Contenu connexe

Similaire à Pwl rewal-slideshare

Similaire à Pwl rewal-slideshare (20)

Dernier

Dernier (20)

Pwl rewal-slideshare

Notes de l'éditeur