As part of our Deep Path Analysis technology, we provide a visualization of the hop-by-hop paths from ThousandEyes agents to a destination site specified by the user. This visualization is very helpful in understanding where packet loss is happening, as well as uncovering segments of the end-to-end path that are routed using labels (Multiprotocol Label Switching- MPLS) instead of IP addresses.
For example, the figure above shows the routes from a ThousandEyes agent in Chicago to the destination node on the extreme right. The label “3” in link adjacent to the agent in the figure means that there are three hops between the Chicago agent and the light blue node on the other end of the link. The white nodes represent interfaces that were not responsive to the agent probes (some router interfaces block ICMP Time Exceeded messages).
The MPLS links can be selected (dashed blue) by clicking on the MPLS quick selection link on the top left. In this case there are 9 links where packets were routed using MPLS labels. MPLS link information can be displayed by mousing over the link, including the MPLS label (L), Experimental flag(E), bottom of Stack (S) and TTL (T) (for more information please refer to RFC4950).
By looking at the path in detail, and coloring red links where delay is higher than 50ms, we observe some interesting behavior.
Node A is the ingress LER (Label Edge Router) of the MPLS tunnel, since it’s the first router in the tunnel to push an MPLS label. Nodes F and G are last hop routers (LH: last routers in the path to pop an MPLS label), and the ones after them are egress LERs. Nodes A, B are located in Chicago, C, D in Denver and E, F and G in San Jose. But wait, how can A and B both be in Chicago, if the latency between them is higher than 50ms?
To answer this question we need to understand what happens when the LSE-TTL of a probe expires. By default latest versions of Cisco IOS do MPLS-based routing of ICMP Time Exceeded packets instead of regular IP based routing (this behavior can be overridden by the mpls ip ttl-expiration pop command). This means time-exceeded packets are first routed to the LH router and only then forward back to the ingress router and back to the source. This is called a u-turn tunnel signature.
In the example above, this means packets expiring downstream of node B would need to travel to F or G before returning to the source. This explains the high delay measured in link A-B: it’s the difference between the delay of a packet that reached A and returned immediately and a packet that reached B, did a u-turn through F (or G) and returned to the source. This also explains why the other links in the MPLS tunnel have pretty small delays - because they all correspond to the same u-turn path, so the delta of delays between consecutive paths are relatively small.
We verified this behaviour when we were analyzing some customer data and were having a hard time correlating geographical locations to latencies between hops. Without having the MPLS route information in each link it would have been almost impossible to verify some of the values of latency observed. Visualize MPLS routes in your corporate network or WAN with a free trial of ThousandEyes.