SlideShare une entreprise Scribd logo
1  sur  25
Télécharger pour lire hors ligne
MPLS RSVP-TE Auto-Bandwidth:
Practical Lessons Learned
1
Richard A Steenbergen <ras@gt-t.net>
MPLS RSVP-TE Quick Review
• MPLS Traffic Engineering 101
• Classically, IGPs used only link cost to select a best path.
• And a “Shortest Path First” (SPF) algorithm finds the “lowest cost” path.
• Traffic Engineering takes this, and adds additional constraints.
• For example, “find the lowest cost path that also has available bandwidth”.
• RSVP-TE accomplishes this by doing the following:
• Measure the amount of bandwidth used between two points in the network.
• Track which circuits the bandwidth is used on, using a reservation system.
• Deny additional reservations when the remaining bandwidth is insufficient.
• And hopefully find a different path which DOES have sufficient bandwidth.
• Bandwidth is measured at the MPLS Label Switched Path (LSP).
• Each LSP is configured with the amount of bandwidth traveling across it.
• The RSVP protocol maps each individual MPLS LSP onto RSVP speaking
circuist, and reserves the amount of bandwidth configured for those LSPs.
2
Example: How to Route from Los Angeles to Chicago
3
Example: How to Route from Los Angeles to Chicago
4
Path 1
Example: How to Route from Los Angeles to Chicago
5
Path 2
Path 1
Example: How to Route from Los Angeles to Chicago
6
Path 2
Path 3
Path 1
Example: How to Route from Los Angeles to Chicago
7
Path 2
Path 3
Path 4
Path 1
MPLS LSP Bandwidth Measurement
• So how do you configure the bandwidth on a particular LSP?
• After all, IP networks are dynamic and packet switched.
• Bandwidth usage can change in and instant, and be unpredictable.
• There are two main ways to accomplish this:
• Offline Calculation
• Calculation which occurs outside of the router, typically based on some
type of “bandwidth modeling”, and often using a third party script or tool.
• This is how RSVP-TE was first implemented, and is still commonly used
today by many large networks and early MPLS adopters.
• Auto-Bandwidth
• The bandwidth value is calculated on the routers themselves, by
periodically measuring how much traffic is actually forwarding over the
LSPs.
8
Offline Calculation vs. Auto-Bandwidth
• Offline Calculation
• You can implement any modeling algorithm you’d like.
• Some extremely complex LSP modeling software is available from a
variety of vendors, allowing you to do very detailed LSP planning.
• But you have to either write the software yourself, or buy it.
• Auto-Bandwidth
• Because it runs directly on the routers, it can respond to changing
traffic conditions much more rapidly, and with less overhead.
• Most offline calculation systems expect very stable traffic patterns.
• Unusual traffic spikes or shifts can cause congestion, or inefficient
bandwidth use.
• Easier to implement (just turn the knob on your router, it’s free).
• But you’re constrained to the algorithms provided by your vendor.
9
LSP Bandwidth Measurement Examples
10
Adjust Interval Adjust Interval Adjust Interval
1.5 Hour Adjust Interval – Leads to more efficient bandwidth use
24 Hour Adjust Interval
How Does Auto-Bandwidth Work?
• Auto-Bandwidth is an entirely router-specific behavior.
• Thus the algorithms can be completely different between vendors.
• It just so happens that both Cisco and Juniper implement it in much
the same way.
• Auto-Bandwidth performs the following basic steps:
1. Every “Statistics Interval”, bandwidth over an LSP is measured.
• For example, you might configure this to every 60 seconds.
1. Every “Adjust Interval”, the largest sample from the process above is
used to calculate the new LSP bandwidth.
• For example, you might use 5 samples and adjust every 300 seconds.
1. If the change is larger than a user configured minimum “Adjust
Threshold”, the new bandwidth value is re-signaled across RSVP.
• Ideally using a make-before-break configuration, to signal the new LSP
with the new bandwidth value before tearing down the old one.
11
When Auto-Bandwidth Works Well
12
When Auto-Bandwidth Doesn’t Work Well
13
Overflow and Underflow
• Another common feature is called “Overflow” and “Underflow”.
• If the difference between the signaled and measured LSP bandwidths
exceeds a configurable percentage for a configurable number of
Statistics Samples, kick off an Adjust event before the normal Adjust
Interval timer would have done so.
• “Overflow” detects increases in traffic rates, “Underflow” detects
decreases in traffic rates.
• The goal is to allow operators to configure a longer Adjust Interval (to
reduce re-signaling), yet still react to rapidly changing traffic conditions.
• WARNING: Some vendors don’t support Underflow.
• Support in IOS XR and JUNOS as of 11.4+
14
Where It All Goes Wrong, A.K.A.
You Knew It Wasn’t Going To Be That Easy
15
RSVP and Changing Traffic Patterns
• RSVP-TE is very good at adapting to changing network
conditions, such as circuit failures or other capacity changes.
• But it is very bad at adapting to changing traffic destinations.
• That is, traffic which was previously destined for router A is now
destined for router B.
• On an IP network, this is typically the result of a change in the BGP
routing. For example:
• An eBGP neighbor flaps on router A, and the traffic moves to router B.
• A circuit flaps, and the resulting IGP cost change modifies the BGP best
path selection, causing traffic to prefer the exit learned from router B.
• Every time this happens, the old LSP must be resized down, and the
new LSP must be resized up. This can take up to an Adjust Interval
amount of time to happen, with potential congestion in the mean time.
16
The Problem With Overflow/Underflow
• Overflow/Underflow can actually cause many problems.
• Consider what happens during a traffic destination shift, where LSP A
must be sized down, and LSP B must be sized up.
• If your router doesn’t support Underflow (e.g. Juniper, Cisco IOS,
etc), you can only react to the increasing LSP, but can’t reclaim the
bandwidth from the newly decreased LSP.
• This creates inefficient routing (or worse) for an entire Adjust Interval
period, which also makes a high Adjust Interval a bad idea.
• They also often don’t create any detection optimization at all.
• If you weren’t going to exceed the minimum threshold of change in
the first place, an RSVP re-signaling would never have occurred.
• You could have just set your Adjust Interval to the lower value to
begin with, for the same results in bandwidth change detection.
17
MPLS LSPs Don’t Just Create Themselves
• Unlike some other protocols, MPLS isn’t entirely automatic.
• There are no protocols to auto-discover MPLS speaking nodes.
• The MPLS “protocols” just exchange label values for the configured LSPs.
• But they have no involvement in the creation of the LSPs themselves.
• Building the full mesh of LSPs is an exercise left to the operator.
• Essentially this means operator supplied scripts are a necessity.
• Or else an operator purchased commercial software solution.
• Examples include WANDL, Cariden, etc.
• Some vendors offer some very basic Auto-Mesh capabilities.
• For example, Cisco IOS can auto-create a mesh of LSPs from a
template, using a list of router IPs supplied via an access-list.
• But this leaves you no way to control a specific LSP configuration.
• Oh, and if you want to remove a node from the mesh, you have to remove
the entire ACL, bringing down every dynamic auto-mesh LSP on the box.
18
Large LSPs Can’t Fit Down Small Pipes
• An LSP can only be moved as an atomic unit.
• So if you have large LSPs relative to the size of the circuits they’re
traveling, efficient packing becomes a serious problem.
• For example, imagine you have 3 x 6 Gbps LSPs and 2 x 10G
circuits.
• 3 x 6 Gbps = 18 Gbps down 20 Gbps of pipe, it should fit right?
• But you’ll actually only be able to fit 2 of the 3 LSPs above.
• The third LSP will have to find another longer path, if one exists at all.
• And your two circuits above will be left with 4 Gbps of unfilled capacity.
• Another example, say you have a mix of 10G and 2.5G circuits:
• A 3 Gbps LSP will never be able to fit down a 2.5G circuit.
• A workaround is to load balance over multiple parallel LSPs
• Instead of having 3 x 6 Gbps LSPs, you could have 9 x 2 Gbps LSPs.
• But so far no router vendor auto-mesh system supports parallel LSPs.
• You can also run into a maximum number of ECMP’d paths limitation. 19
Auto-Bandwidth Behavior Under Stress
• Another issue is the behavior of auto-bandwidth systems
when RSVP cannot find sufficient bandwidth on any link.
• For example, consider the previous example of poor LSP packing.
• Sometimes, when a new bandwidth reservation cannot be met, the
LSP value is simple not updated even though traffic has increased.
• This leads to non-RSVP accounted traffic on the network, which often
causes silent congestion where RSVP thinks there should be none.
• Under other conditions, an updated bandwidth reservation which
cannot be met causes an existing LSP to be torn down completely.
• In a parallel-LSP scenario, traffic immediately shifts to the remaining LSPs.
• In the short term, non-accounted traffic is now traveling over these links.
• But even worse, the remaining LSPs have now become larger, and are
also likely to fail to find sufficient bandwidth. This leads to a pathological
behavior where all of the parallel LSPs fail, resulting in a single large LSP
which can never find bandwidth, without manual human intervention.
20
What About An Intelligent Auto-Mesh Script?
• Imagine you have to implement your own LSP Auto-Mesh script
anyways, to support configuring multiple parallel LSPs.
• Could you make it automatically “fork” LSPs which get too big?
• For example, this could be done automatically with a Juniper Event Script.
• When a sufficiently large LSP is detected, the router could automatically
configure a new parallel LSP.
• But not so fast, there are gotchas here too:
• Newly created auto-bandwidth LSPs initially signal with a bandwidth value
of 0, even though traffic is immediately load-balanced over them.
• And there is no way to override this with current Cisco/Juniper implementations.
• This leads to incorrect RSVP bandwidth measures, non-RSVP accounted
traffic on the network, and congestion often results.
• And under the previously explained “pathological LSP collapse” scenario,
creating a pile of new LSPs would make the situation even worse.
21
Auto-Bandwidth and Congestion
• Auto-Bandwidth doesn’t know anything about congestion.
• Imagine that for some reason (such as the many examples provided)
a link becomes congested, even though RSVP is unaware of it.
• Packet loss causes TCP to throttle back, and the IP traffic goes down.
• Auto-Bandwidth adapts to this new rate, and thinks everything is fine.
• This can lead to sustained congestion, requiring manual intervention.
• Also be careful of routers which can’t “see” layer 2 overhead.
• For example, Juniper M/T/MX series routers can’t measure any of it.
• But every vendor misses at least some of the L2 per-packet overhead.
• A 28-byte UDP packet consumes 84 bytes over the wire on Ethernet.
• But if your router can’t measure this, a Denial of Service attack (or
even a shift of traffic with small packets, such as TCP ACKs) can
cause a link to become congested even though RSVP thinks it’s fine.
22
Some Practical Suggestions
• Well designed RSVP-TE Auto-Bandwidth can work well.
• In fact, most of the time it works great.
• But a wide variety of conditions exist where human intervention is
required to resolve pathological issues.
• Don’t bother using Overflow
• Especially if you don’t also have Underflow.
• Beware of vendor bugs.
• We’ve seen MANY conditions under which auto-bandwidth
measurements can be affected by router vendor bugs.
• Careful monitoring of your network is still required.
23
Nag Your Vendors For
• Don’t forget to nag your vendors for:
• An adjust-threshold minimum in bytes as well as percent!
• So you don’t have to re-signal every LSP that changes from 256Kbps to
512Kbps every adjust-interval.
• Better built-in LSP Auto-Mesh capabilities.
• Underflow as well as Overflow support, if you really want to use it.
• A feature to automatically “fork” large LSPs past a certain size.
• A better way to display these forked LSPs in your “show route”
output so you don’t end up with a screen full of LSP next-hops on
every route when using parallel LSPs.
• The ability to set an “initial bandwidth reservation” for new LSPs.
• An operational mode command to manually adjust the bandwidth
values of an LSP, if an operator supplied auto-mesh script is
required to implement LSP forking.
24
Send questions, comments, complaints to:
Richard A Steenbergen <ras@gt-t.net>

Contenu connexe

Tendances

RIPE 76: TCP and BBR
RIPE 76: TCP and BBRRIPE 76: TCP and BBR
RIPE 76: TCP and BBRAPNIC
 
NZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBRNZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBRAPNIC
 
AusNOG 2019: TCP and BBR
AusNOG 2019: TCP and BBRAusNOG 2019: TCP and BBR
AusNOG 2019: TCP and BBRAPNIC
 
Traffic profiles, congestion and network performance
Traffic profiles, congestion and network performanceTraffic profiles, congestion and network performance
Traffic profiles, congestion and network performanceRaj Parekh
 
Congestion control
Congestion controlCongestion control
Congestion controlNithin Raj
 
Congestion avoidance in TCP
Congestion avoidance in TCPCongestion avoidance in TCP
Congestion avoidance in TCPselvakumar_b1985
 
Congestion on computer network
Congestion on computer networkCongestion on computer network
Congestion on computer networkDisi Dc
 
RIPE 80: Buffers and Protocols
RIPE 80: Buffers and ProtocolsRIPE 80: Buffers and Protocols
RIPE 80: Buffers and ProtocolsAPNIC
 
The Stories of IXP Development and the Way Forward by Che-Hoo Cheng
The Stories of IXP Development and the Way Forward by Che-Hoo ChengThe Stories of IXP Development and the Way Forward by Che-Hoo Cheng
The Stories of IXP Development and the Way Forward by Che-Hoo ChengMyNOG
 
TCP and BBR
TCP and BBRTCP and BBR
TCP and BBRAPNIC
 
How Data Center Traffic is Changing Your Network by KC Lim
How Data Center Traffic is Changing Your Network by KC LimHow Data Center Traffic is Changing Your Network by KC Lim
How Data Center Traffic is Changing Your Network by KC LimMyNOG
 
IPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd RamlyIPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd RamlyMyNOG
 

Tendances (20)

RIPE 76: TCP and BBR
RIPE 76: TCP and BBRRIPE 76: TCP and BBR
RIPE 76: TCP and BBR
 
NZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBRNZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBR
 
AusNOG 2019: TCP and BBR
AusNOG 2019: TCP and BBRAusNOG 2019: TCP and BBR
AusNOG 2019: TCP and BBR
 
Traffic profiles, congestion and network performance
Traffic profiles, congestion and network performanceTraffic profiles, congestion and network performance
Traffic profiles, congestion and network performance
 
Congestion control
Congestion controlCongestion control
Congestion control
 
Congestion control
Congestion controlCongestion control
Congestion control
 
Leaky bucket A
Leaky bucket ALeaky bucket A
Leaky bucket A
 
Congestion avoidance in TCP
Congestion avoidance in TCPCongestion avoidance in TCP
Congestion avoidance in TCP
 
Congestion on computer network
Congestion on computer networkCongestion on computer network
Congestion on computer network
 
Congestion control
Congestion controlCongestion control
Congestion control
 
RIPE 80: Buffers and Protocols
RIPE 80: Buffers and ProtocolsRIPE 80: Buffers and Protocols
RIPE 80: Buffers and Protocols
 
Conjestion control
Conjestion controlConjestion control
Conjestion control
 
Congestion control
Congestion controlCongestion control
Congestion control
 
Congestion Control
Congestion ControlCongestion Control
Congestion Control
 
The Stories of IXP Development and the Way Forward by Che-Hoo Cheng
The Stories of IXP Development and the Way Forward by Che-Hoo ChengThe Stories of IXP Development and the Way Forward by Che-Hoo Cheng
The Stories of IXP Development and the Way Forward by Che-Hoo Cheng
 
TCP and BBR
TCP and BBRTCP and BBR
TCP and BBR
 
Tcp(no ip) review part1
Tcp(no ip) review part1Tcp(no ip) review part1
Tcp(no ip) review part1
 
How Data Center Traffic is Changing Your Network by KC Lim
How Data Center Traffic is Changing Your Network by KC LimHow Data Center Traffic is Changing Your Network by KC Lim
How Data Center Traffic is Changing Your Network by KC Lim
 
IPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd RamlyIPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
 
Tcp(no ip) review part2
Tcp(no ip) review part2Tcp(no ip) review part2
Tcp(no ip) review part2
 

Similaire à MPLS RSVP-TE Auto-Bandwidth - Practical Lessons Learned

JPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay Nodes
JPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay NodesJPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay Nodes
JPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay Nodeschennaijp
 
Advanced Traffic Engineering (TE++)
Advanced Traffic Engineering (TE++)Advanced Traffic Engineering (TE++)
Advanced Traffic Engineering (TE++)Pravin Bhandarkar
 
Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and Hadoop
Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and HadoopEventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and Hadoop
Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and HadoopAyon Sinha
 
Computer networks unit iii
Computer networks    unit iiiComputer networks    unit iii
Computer networks unit iiiJAIGANESH SEKAR
 
Layer3protocols
Layer3protocolsLayer3protocols
Layer3protocolsassinha
 
Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...
Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...
Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...Continuent
 
Routing, Network Performance, and Role of Analytics
Routing, Network Performance, and Role of AnalyticsRouting, Network Performance, and Role of Analytics
Routing, Network Performance, and Role of AnalyticsAPNIC
 
2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...
2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...
2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...IEEEFINALYEARSTUDENTPROJECT
 
2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...
2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...
2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...IEEEFINALSEMSTUDENTSPROJECTS
 
IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...
IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...
IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...IEEEGLOBALSOFTSTUDENTPROJECTS
 
ROUTING PROTOCOLS new.pptx
ROUTING PROTOCOLS new.pptxROUTING PROTOCOLS new.pptx
ROUTING PROTOCOLS new.pptxAayushMishra89
 
PLNOG 3: John Evans - Best Practices in Network Planning
PLNOG 3: John Evans - Best Practices in Network PlanningPLNOG 3: John Evans - Best Practices in Network Planning
PLNOG 3: John Evans - Best Practices in Network PlanningPROIDEA
 
IETF87 - STATUS BoF: Performance Engineered LSPs
IETF87 - STATUS BoF: Performance Engineered LSPsIETF87 - STATUS BoF: Performance Engineered LSPs
IETF87 - STATUS BoF: Performance Engineered LSPsRob Shakir
 
Routing In Fat Trees
Routing In Fat TreesRouting In Fat Trees
Routing In Fat TreesAPNIC
 

Similaire à MPLS RSVP-TE Auto-Bandwidth - Practical Lessons Learned (20)

Introduction to MPLS - NANOG 61
Introduction to MPLS - NANOG 61Introduction to MPLS - NANOG 61
Introduction to MPLS - NANOG 61
 
JPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay Nodes
JPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay NodesJPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay Nodes
JPJ1433 Cost-Effective Resource Allocation of Overlay Routing Relay Nodes
 
Advanced Traffic Engineering (TE++)
Advanced Traffic Engineering (TE++)Advanced Traffic Engineering (TE++)
Advanced Traffic Engineering (TE++)
 
Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and Hadoop
Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and HadoopEventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and Hadoop
Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and Hadoop
 
Computer networks unit iii
Computer networks    unit iiiComputer networks    unit iii
Computer networks unit iii
 
Layer3protocols
Layer3protocolsLayer3protocols
Layer3protocols
 
Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...
Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...
Webinar Slides: Tungsten Connector / Proxy – The Secret Sauce Behind Zero-Dow...
 
Routing, Network Performance, and Role of Analytics
Routing, Network Performance, and Role of AnalyticsRouting, Network Performance, and Role of Analytics
Routing, Network Performance, and Role of Analytics
 
2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...
2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...
2014 IEEE JAVA NETWORKING COMPUTING PROJECT Cost effective resource allocatio...
 
2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...
2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...
2014 IEEE JAVA NETWORKING PROJECT Cost effective resource allocation of overl...
 
IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...
IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...
IEEE 2014 JAVA NETWORKING PROJECTS Cost effective resource allocation of over...
 
Chap2 slides
Chap2 slidesChap2 slides
Chap2 slides
 
ROUTING PROTOCOLS new.pptx
ROUTING PROTOCOLS new.pptxROUTING PROTOCOLS new.pptx
ROUTING PROTOCOLS new.pptx
 
PLNOG 3: John Evans - Best Practices in Network Planning
PLNOG 3: John Evans - Best Practices in Network PlanningPLNOG 3: John Evans - Best Practices in Network Planning
PLNOG 3: John Evans - Best Practices in Network Planning
 
Ospf
OspfOspf
Ospf
 
SPROJReport (1)
SPROJReport (1)SPROJReport (1)
SPROJReport (1)
 
IETF87 - STATUS BoF: Performance Engineered LSPs
IETF87 - STATUS BoF: Performance Engineered LSPsIETF87 - STATUS BoF: Performance Engineered LSPs
IETF87 - STATUS BoF: Performance Engineered LSPs
 
Advanced networking scheduling and QoS part 2
Advanced networking scheduling and QoS part 2Advanced networking scheduling and QoS part 2
Advanced networking scheduling and QoS part 2
 
6978106.ppt
6978106.ppt6978106.ppt
6978106.ppt
 
Routing In Fat Trees
Routing In Fat TreesRouting In Fat Trees
Routing In Fat Trees
 

Dernier

TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSedrianrheine
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpressssuser166378
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Shubham Pant
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilitiesalihassaah1994
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSlesteraporado16
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdfShreedeep Rayamajhi
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxnaveenithkrishnan
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...APNIC
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024Jan Löffler
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteMavein
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsRoxana Stingu
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfmchristianalwyn
 

Dernier (12)

TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpress
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilities
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptx
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a Website
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
 

MPLS RSVP-TE Auto-Bandwidth - Practical Lessons Learned

  • 1. MPLS RSVP-TE Auto-Bandwidth: Practical Lessons Learned 1 Richard A Steenbergen <ras@gt-t.net>
  • 2. MPLS RSVP-TE Quick Review • MPLS Traffic Engineering 101 • Classically, IGPs used only link cost to select a best path. • And a “Shortest Path First” (SPF) algorithm finds the “lowest cost” path. • Traffic Engineering takes this, and adds additional constraints. • For example, “find the lowest cost path that also has available bandwidth”. • RSVP-TE accomplishes this by doing the following: • Measure the amount of bandwidth used between two points in the network. • Track which circuits the bandwidth is used on, using a reservation system. • Deny additional reservations when the remaining bandwidth is insufficient. • And hopefully find a different path which DOES have sufficient bandwidth. • Bandwidth is measured at the MPLS Label Switched Path (LSP). • Each LSP is configured with the amount of bandwidth traveling across it. • The RSVP protocol maps each individual MPLS LSP onto RSVP speaking circuist, and reserves the amount of bandwidth configured for those LSPs. 2
  • 3. Example: How to Route from Los Angeles to Chicago 3
  • 4. Example: How to Route from Los Angeles to Chicago 4 Path 1
  • 5. Example: How to Route from Los Angeles to Chicago 5 Path 2 Path 1
  • 6. Example: How to Route from Los Angeles to Chicago 6 Path 2 Path 3 Path 1
  • 7. Example: How to Route from Los Angeles to Chicago 7 Path 2 Path 3 Path 4 Path 1
  • 8. MPLS LSP Bandwidth Measurement • So how do you configure the bandwidth on a particular LSP? • After all, IP networks are dynamic and packet switched. • Bandwidth usage can change in and instant, and be unpredictable. • There are two main ways to accomplish this: • Offline Calculation • Calculation which occurs outside of the router, typically based on some type of “bandwidth modeling”, and often using a third party script or tool. • This is how RSVP-TE was first implemented, and is still commonly used today by many large networks and early MPLS adopters. • Auto-Bandwidth • The bandwidth value is calculated on the routers themselves, by periodically measuring how much traffic is actually forwarding over the LSPs. 8
  • 9. Offline Calculation vs. Auto-Bandwidth • Offline Calculation • You can implement any modeling algorithm you’d like. • Some extremely complex LSP modeling software is available from a variety of vendors, allowing you to do very detailed LSP planning. • But you have to either write the software yourself, or buy it. • Auto-Bandwidth • Because it runs directly on the routers, it can respond to changing traffic conditions much more rapidly, and with less overhead. • Most offline calculation systems expect very stable traffic patterns. • Unusual traffic spikes or shifts can cause congestion, or inefficient bandwidth use. • Easier to implement (just turn the knob on your router, it’s free). • But you’re constrained to the algorithms provided by your vendor. 9
  • 10. LSP Bandwidth Measurement Examples 10 Adjust Interval Adjust Interval Adjust Interval 1.5 Hour Adjust Interval – Leads to more efficient bandwidth use 24 Hour Adjust Interval
  • 11. How Does Auto-Bandwidth Work? • Auto-Bandwidth is an entirely router-specific behavior. • Thus the algorithms can be completely different between vendors. • It just so happens that both Cisco and Juniper implement it in much the same way. • Auto-Bandwidth performs the following basic steps: 1. Every “Statistics Interval”, bandwidth over an LSP is measured. • For example, you might configure this to every 60 seconds. 1. Every “Adjust Interval”, the largest sample from the process above is used to calculate the new LSP bandwidth. • For example, you might use 5 samples and adjust every 300 seconds. 1. If the change is larger than a user configured minimum “Adjust Threshold”, the new bandwidth value is re-signaled across RSVP. • Ideally using a make-before-break configuration, to signal the new LSP with the new bandwidth value before tearing down the old one. 11
  • 14. Overflow and Underflow • Another common feature is called “Overflow” and “Underflow”. • If the difference between the signaled and measured LSP bandwidths exceeds a configurable percentage for a configurable number of Statistics Samples, kick off an Adjust event before the normal Adjust Interval timer would have done so. • “Overflow” detects increases in traffic rates, “Underflow” detects decreases in traffic rates. • The goal is to allow operators to configure a longer Adjust Interval (to reduce re-signaling), yet still react to rapidly changing traffic conditions. • WARNING: Some vendors don’t support Underflow. • Support in IOS XR and JUNOS as of 11.4+ 14
  • 15. Where It All Goes Wrong, A.K.A. You Knew It Wasn’t Going To Be That Easy 15
  • 16. RSVP and Changing Traffic Patterns • RSVP-TE is very good at adapting to changing network conditions, such as circuit failures or other capacity changes. • But it is very bad at adapting to changing traffic destinations. • That is, traffic which was previously destined for router A is now destined for router B. • On an IP network, this is typically the result of a change in the BGP routing. For example: • An eBGP neighbor flaps on router A, and the traffic moves to router B. • A circuit flaps, and the resulting IGP cost change modifies the BGP best path selection, causing traffic to prefer the exit learned from router B. • Every time this happens, the old LSP must be resized down, and the new LSP must be resized up. This can take up to an Adjust Interval amount of time to happen, with potential congestion in the mean time. 16
  • 17. The Problem With Overflow/Underflow • Overflow/Underflow can actually cause many problems. • Consider what happens during a traffic destination shift, where LSP A must be sized down, and LSP B must be sized up. • If your router doesn’t support Underflow (e.g. Juniper, Cisco IOS, etc), you can only react to the increasing LSP, but can’t reclaim the bandwidth from the newly decreased LSP. • This creates inefficient routing (or worse) for an entire Adjust Interval period, which also makes a high Adjust Interval a bad idea. • They also often don’t create any detection optimization at all. • If you weren’t going to exceed the minimum threshold of change in the first place, an RSVP re-signaling would never have occurred. • You could have just set your Adjust Interval to the lower value to begin with, for the same results in bandwidth change detection. 17
  • 18. MPLS LSPs Don’t Just Create Themselves • Unlike some other protocols, MPLS isn’t entirely automatic. • There are no protocols to auto-discover MPLS speaking nodes. • The MPLS “protocols” just exchange label values for the configured LSPs. • But they have no involvement in the creation of the LSPs themselves. • Building the full mesh of LSPs is an exercise left to the operator. • Essentially this means operator supplied scripts are a necessity. • Or else an operator purchased commercial software solution. • Examples include WANDL, Cariden, etc. • Some vendors offer some very basic Auto-Mesh capabilities. • For example, Cisco IOS can auto-create a mesh of LSPs from a template, using a list of router IPs supplied via an access-list. • But this leaves you no way to control a specific LSP configuration. • Oh, and if you want to remove a node from the mesh, you have to remove the entire ACL, bringing down every dynamic auto-mesh LSP on the box. 18
  • 19. Large LSPs Can’t Fit Down Small Pipes • An LSP can only be moved as an atomic unit. • So if you have large LSPs relative to the size of the circuits they’re traveling, efficient packing becomes a serious problem. • For example, imagine you have 3 x 6 Gbps LSPs and 2 x 10G circuits. • 3 x 6 Gbps = 18 Gbps down 20 Gbps of pipe, it should fit right? • But you’ll actually only be able to fit 2 of the 3 LSPs above. • The third LSP will have to find another longer path, if one exists at all. • And your two circuits above will be left with 4 Gbps of unfilled capacity. • Another example, say you have a mix of 10G and 2.5G circuits: • A 3 Gbps LSP will never be able to fit down a 2.5G circuit. • A workaround is to load balance over multiple parallel LSPs • Instead of having 3 x 6 Gbps LSPs, you could have 9 x 2 Gbps LSPs. • But so far no router vendor auto-mesh system supports parallel LSPs. • You can also run into a maximum number of ECMP’d paths limitation. 19
  • 20. Auto-Bandwidth Behavior Under Stress • Another issue is the behavior of auto-bandwidth systems when RSVP cannot find sufficient bandwidth on any link. • For example, consider the previous example of poor LSP packing. • Sometimes, when a new bandwidth reservation cannot be met, the LSP value is simple not updated even though traffic has increased. • This leads to non-RSVP accounted traffic on the network, which often causes silent congestion where RSVP thinks there should be none. • Under other conditions, an updated bandwidth reservation which cannot be met causes an existing LSP to be torn down completely. • In a parallel-LSP scenario, traffic immediately shifts to the remaining LSPs. • In the short term, non-accounted traffic is now traveling over these links. • But even worse, the remaining LSPs have now become larger, and are also likely to fail to find sufficient bandwidth. This leads to a pathological behavior where all of the parallel LSPs fail, resulting in a single large LSP which can never find bandwidth, without manual human intervention. 20
  • 21. What About An Intelligent Auto-Mesh Script? • Imagine you have to implement your own LSP Auto-Mesh script anyways, to support configuring multiple parallel LSPs. • Could you make it automatically “fork” LSPs which get too big? • For example, this could be done automatically with a Juniper Event Script. • When a sufficiently large LSP is detected, the router could automatically configure a new parallel LSP. • But not so fast, there are gotchas here too: • Newly created auto-bandwidth LSPs initially signal with a bandwidth value of 0, even though traffic is immediately load-balanced over them. • And there is no way to override this with current Cisco/Juniper implementations. • This leads to incorrect RSVP bandwidth measures, non-RSVP accounted traffic on the network, and congestion often results. • And under the previously explained “pathological LSP collapse” scenario, creating a pile of new LSPs would make the situation even worse. 21
  • 22. Auto-Bandwidth and Congestion • Auto-Bandwidth doesn’t know anything about congestion. • Imagine that for some reason (such as the many examples provided) a link becomes congested, even though RSVP is unaware of it. • Packet loss causes TCP to throttle back, and the IP traffic goes down. • Auto-Bandwidth adapts to this new rate, and thinks everything is fine. • This can lead to sustained congestion, requiring manual intervention. • Also be careful of routers which can’t “see” layer 2 overhead. • For example, Juniper M/T/MX series routers can’t measure any of it. • But every vendor misses at least some of the L2 per-packet overhead. • A 28-byte UDP packet consumes 84 bytes over the wire on Ethernet. • But if your router can’t measure this, a Denial of Service attack (or even a shift of traffic with small packets, such as TCP ACKs) can cause a link to become congested even though RSVP thinks it’s fine. 22
  • 23. Some Practical Suggestions • Well designed RSVP-TE Auto-Bandwidth can work well. • In fact, most of the time it works great. • But a wide variety of conditions exist where human intervention is required to resolve pathological issues. • Don’t bother using Overflow • Especially if you don’t also have Underflow. • Beware of vendor bugs. • We’ve seen MANY conditions under which auto-bandwidth measurements can be affected by router vendor bugs. • Careful monitoring of your network is still required. 23
  • 24. Nag Your Vendors For • Don’t forget to nag your vendors for: • An adjust-threshold minimum in bytes as well as percent! • So you don’t have to re-signal every LSP that changes from 256Kbps to 512Kbps every adjust-interval. • Better built-in LSP Auto-Mesh capabilities. • Underflow as well as Overflow support, if you really want to use it. • A feature to automatically “fork” large LSPs past a certain size. • A better way to display these forked LSPs in your “show route” output so you don’t end up with a screen full of LSP next-hops on every route when using parallel LSPs. • The ability to set an “initial bandwidth reservation” for new LSPs. • An operational mode command to manually adjust the bandwidth values of an LSP, if an operator supplied auto-mesh script is required to implement LSP forking. 24
  • 25. Send questions, comments, complaints to: Richard A Steenbergen <ras@gt-t.net>