Investigation of Traffic Dependencies between IXPs in Failure Scenarios, by Arnold Nipper. A presentation given at APRICOT 2016’s Peering Forum (2) session on 23 February 2016.

1. Inves&ga&on
of
Traffic
Dependencies
between
IXPs
in
Failure
Scenarios
APRICOT
2016,
Peering
Forum
2016-­‐02-­‐23
Auckland,
New
Zealand
Arnold
Nipper
Chief
Technology
Evangelist
and
Co-­‐Founder

2. » How
robust
is
the
IXP
interconnecKon
system?
» What
happens
if
a
large
IXP
fails?
» Does
it
affect
other
IXPs
and
how?
» There
was
an
incident,
which
we
invesKgated
» This
presentaKon
is
about
the
results
» What
can
we
learn
from
this?
Introduc&on
2

3. Incident
at
a
large
IXP
in
Europe
» 13th
May
2015
at
12:22
pm
» Loop
with
4
x
100GE,
traffic
was
blackholed.
» About
500
of
600
BGP
session
drops
at
the
route
servers
3

4. Which
impact
would
you
expect
on
other
IXPs?
4
?

5. » Decreased
traffic
volume
» Drop
of
about
240
Gbps
within
5
minutes
» Recovering
a[er
about
10
minutes
Impact
on
DE-­‐CIX
Frankfurt
2.26
Tbps
2.50
Tbps
5

6. 13th
May
2015:
(informaKon
from
public
sources)
1. 12:22
pm
–
Loop
with
4
x
100GE
created.
Traffic
was
blackholed.
2. 12:25
pm
–
About
500
of
600
BGP
sessions
at
the
route
servers
dropped
3. 12:29
pm
–
NOC
reacted
and
deacKvated
ports
responsible
for
loop
4. 12:40
pm
–
BGP
sessions
to
route
server
are
back
online
Impact
on
DE-­‐CIX
Frankfurt
-­‐-­‐
Time
Flow
1.
2.
3.
4.
Large
European
IXP
DE-­‐CIX
6

7. What
could
be
the
reason
for
this
behavior?
What
are
dependencies?
We
found
three
answers…
so
far…
7

8. 1.
Remote
Peering
Routers
Overloaded
» Overloaded
remote
peering
router
drops
all
BGP
sessions
» Four
customers
at
DE-­‐CIX
Frankfurt
affected
with
a
traffic
volume
drop
of
0.92
Gbit/s
8
IXP
A
IXP
B
Remote
peering
router

9. 2.
Asymmetric
Rou&ng
Paths
9
Is
there
a
significant
number
of
asymmetric
paths
traversing
both
IXPs?
Example:
» Upstream
traverses
IXP
A
» Downstream
traverses
IXP
B
AS
1
AS
2
IXP
A
IXP
B

10. RIPE Atlas measurements:
» ASes connected to DE-CIX Frankfurt and AMS-IX Amsterdam: 323 (40-50%)
» “Peerings” with a traffic drop > 200Mbit/s at DE-CIX Frankfurt: 183
» ASes hosting (at least one) RIPE Atlas probe: 171
» Intersection: 50 AS-to-AS connections
Measurement results of full mesh traceroutes:
» 38% of all AS-to-AS connection with at least one asymmetric IXP path
» 8% of all AS-to-AS connection traversed no IXP
» Impact on traffic drop
» 29,8 Gbit/s due to potential asymmetric routing
2.
Asymmetric
Rou&ng
Paths
II
10

11. 2. Asymmetric Routing Paths
15.65%
49.89%
34.46%
8.01%
59.73%
32.26%
8.52%
63.96%
27.52%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Asymmetric
No-­‐IXP
Other
DE-­‐CIX
/
AMSIX
DE-­‐CIX
/
LINX
AMSIX
/
LINX

12. Impact
Details
12
Shares
in
Mbit/s

13. » Users
experienced
connecKon
errors
» Users
were
annoyed
by
broken
“Internet”
and
switched
acKviKes
» Less
users
resulted
in
less
traffic
» Impact
on
traffic
volume
is
hard
to
quanKfy
3.
Layer
8:
Less
Users
13

14. 14
Ac&ve
Measurement
during
Outage
hips://labs.ripe.net/Members/emileaben/does-­‐the-­‐internet-­‐route-­‐around-­‐damage
��
���
���
���
���
���� ������
������
������
������
������
������
������
������
������
�����������������������
���� �� ��� �����
������������ ������� �������� ������������������ �����
�� ���� �� ���� ����� �����������
������������ ����
������������ ��� ����
�������� ��� ����
�������� ����

15. Reasons
for
traffic
volume
dependencies
between
IXPs:
1. Remote
peering
routers
overloaded
2. Asymmetric
rouKng
paths
3. Layer
8:
Less
users
Good
news:
The
Internet
infrastructure
is
not
hit
largely
if
a
large
IXP
fails.
Takeaway:
» Knowledge
of
traffic
dependencies
of
IXPs
» Useful
for
designing
peering
and
especially
remote
peering
» Improve
recovery
Kme
e.g.
route
server
BFD
Summary
and
Takeaway
15

16. DE-CIX Management GmbH
Lindleystr. 12
60314 Frankfurt
Germany
Phone +49 69 1730 902 0
rnd@de-cix.net