Digital Experience Assurance (DXA) is about understanding whether the products and services you have built, designed, and are responsible for consistently deliver good value to your users, customers, and partners. It is about knowing that all the links of this hugely complex, end-to-end digital supply chain are intact, strong, and, quite simply, performing optimally and fully completing their duties. It is about more than resolving a fault—it’s about assuring that there is consistency in the performance of that service delivery.
ThousandEyes’ DXA provides IT executive leadership with the confidence and ongoing support to allow the gears to always keep spinning in their operations, avoid or minimize the unpleasantly high costs of downtime or performance impacts, and protect the reputation of the organization.
A fundamental paradigm of DXA is knowing exactly what, where, and why a performance issue is developing or has occurred—this includes the ability to know that all the granular mechanisms are doing their part, as designed and intended, including key control plane functions such as BGP and critical data plane measurements such as Maximum Transmission Unit.
Exploring these topics further, below is a fantastic excerpt from Cisco ThousandEyes: Digital Experience Monitoring and Troubleshooting (Cisco Press), a book authored by Aaron Trompeter and Robert Webb.
MTU Best Practices
Maximum Transmission Unit (MTU) issues can lead to performance and reliability issues that are difficult to isolate. Determining root cause on MTU issues often comes down to three things:
- Defining and standardizing MTUs in the network architecture
- Implementing those standards
- Confirming those standards were implemented properly
This is why it is highly recommended to architect consistent MTUs. By defining one MTU for all overlay tunnels throughout the entire network and another for underlay networks throughout the entire network. On a single enterprise network, every tunnel should be set to the same MTU. Ideally Path MTU discovery (PMTUD) is operational in the environment (ICMP Type 3 Code 4 are allowed). Whether it is or not, a consistent MTU should be defined and built into each tunnel interface as a standard practice. As an example, if one interface requires an MTU of 1400, but another requires an MTU of 1372, then the smaller of the MTUs should become the standard for all interfaces. However, the justification for why the MTUs would require different values in the first place should be fully understood. An example might be that there are different encryption methods used for different tunnels. Regardless, the smallest required value should be implemented throughout the organization as the standard. This will allow systems and applications that do not use PMTUD (DF bit = 0) or that may have PMTUD blocked, to avoid fragmentation issues.
The underlay should be treated the same way. While the underlay may require different MTUs than the overlay, all underlay links should relay on the same MTU value. If this is something other than 1500, the smallest value should be selected as the standard. Often, underlay networks utilize the Internet or a third party provider to supply network connectivity. In which case, MTUs are defined outside of your organization. In this case, testing MTUs between locations is critical. Adapting the edge routers to use the smallest MTU may be the best approach. This will avoid fragmentation when different MTUs are discovered across a common path.
If a system does not use PMTUD, then it is likely setting the Don’t Fragment (DF) bit in the IP header to 0. This tells routers in the path that it is ok to fragment any packets that are larger than the MTU on an interface to make the packets fit through that interface/tunnel. If two different MTUs exist in the same path, this leads to double fragmentation which some applications may fail to reassemble. Likewise, having MTU values that differ depending upon the direction of traffic can lead to strange fragmentation issues or even traffic failing to be delivered at all.
Overlay
Figure 1 below shows an overlay test running between Agent 1 and Agent 2. From Agent 1 to Agent 2 (source to target) the MTU is 1362. However, when testing from Agent 2 back to Agent 1 (target to source) now the MTU is 1363. These MTUs should be set to the same value.
Figure 2 shows what happens when Agent 2 sends data, in this case Ping Requests, to Agent 1 at each of those MTU values. Keep in mind, Agent 2 believes (because of PMTUD) that the MTU to Agent 1 is 1363. In packets 5646 through 5718, we see Pings (Echo Request) with a total length (layer 3) of 1362 being replied to with Echo Replies also with a length of 1362. However, beginning with packet 6301, we see a Echo Request with a length of 1363 being issues which results in a fragmented reply where the first packet is 1362 in length. This is because Agent 1 believes the MTU to Agent 2 is set to 1362.
Figure 3 shows the path from Agents 1 and 2 to a Microsoft Active Directory server. One path has an MTU of 1371 while the other path shows and MTU of 1362. Once again, standardizing on the smallest required MTU would not only simplify the process of defining the architecture, it would also simplify implementing it and troubleshooting it.
Underlay
Figure 4 shows the underlay path from Agent 1 to a public IP address on a router located at the site of Agent 2. In this path, one MTU is set to 1492 as it leaves the customer network and enters the Internet. Later we see another MTU set to 1476 as it arrives at its destination. If the underlay requires an MTU of 1476, all non-default (1500) MTU interfaces should share the value of 1476 (or whatever is the smallest required MTU for your company’s underlay network).
In summary, it is highly recommended that these MTU values be applied consistently across the Overlay and the Underlay networks. Each having its own value. Once standardized MTUs are in place, ThousandEyes can be configured to alert of any MTUs that are seen to be smaller than the standards chosen.
Alerting
Figure 5 shows an example of how an overlay alert rule could be created to identify whether any MTU other than 1362 is discovered on the network. This would be ideal to help in change validation to ensure the correct MTUs are implemented. Note that you would not want to deploy this alert rule on any test that is either running in the underlay or not going through an overlay tunnel. Make sure you only include tests and agents that will be impacted by the smaller MTU. If you have agents configured in the test that will stay at 1500 bytes, you will need to include or exclude agents from this alert, whichever is easier for your implementation.
Figure 6 is showing an example of alerting on underlay paths that an MTU of other than 1472.
Conclusion
MTU is a crucial network variable that must be accounted for and proactively monitored for misconfigurations, changes, or suboptimal settings along the data path. Improperly assigned MTU values can wreak havoc in ways that can be painfully challenging to isolate and identify as the root cause, resulting in non-optimal IP fragmentation or, even worse, dropping the datagram, especially in complex application flows that traverse several networks. Accounting for MTU, alongside the many other service delivery affecting parameters in an end-to-end path, is absolutely critical.
Today’s digital services weave through a mind-bogglingly complex array of networks, domains, systems, and physical hardware. From the edge to the WAN to the Internet to the many providers of network transport, infrastructure, and cloud services, knowing what to look at, how to look at it, and how to tie it all together into a bigger picture can be overwhelming. ThousandEyes enables organizations to see every network to assure every experience, facilitating real-time issue detection, confident remediation, and continuous optimization for every connected experience.
Unlock 35%* savings on Cisco ThousandEyes: Digital Experience Monitoring and Troubleshooting! Use code THOUSANDEYES at checkout to save on the book or eBook. Order now from Cisco Press for free U.S. shipping and multi-format eBooks.
*Discount code THOUSANDEYES confers a 35% discount on ISBNs 9780138308971 and 9780138309183 only. Apply the discount code during checkout to receive savings. This discount is not valid on "Best Value" or "Additional Savings" book + eBook bundles, practice tests, video courses, Rough Cuts, O'Reilly Online Learning subscriptions, or simulator software. The discount code may not be combined with any other offer and is not redeemable for cash. This discount offer expires at 11:59 p.m. EDT on December 31, 2024. Offer subject to change.