ANNOUNCEMENT
Cisco ThousandEyes Achieves FedRAMP® Moderate Authorization

Engineering

Friendship, Feuds, and Fiber: The Business of Peering

By Engineering Team
| | 13 min read

Summary

The Internet is less a machine than a treaty system with fiber optics. It’s not just cables and protocols—it’s alliances, trade routes, and the occasional border standoff, with caffeine-fueled engineers acting as diplomats so your data can slip through like it owns the place.


The Postal System of the Internet

In the mid-19th century, sending a letter from, say, Switzerland to Portugal could feel less like “communication” and more like a live-action board game. You didn’t just slap on a stamp and drop it in a box. You often had to navigate a patchwork of postal treaties, prepay multiple postal administrations for transit rights, and hope each border crossing honored the previous country’s arrangements. If the paperwork wasn’t perfect, your letter got stuck in postal limbo—returned to sender, held for additional fees, or delayed indefinitely while postal clerks argued over jurisdiction. This wasn’t a technology problem—horses knew how to walk, trains knew how to roll, ships knew how to float—but an administrative and economic one. That’s how you ended up with envelopes covered in multiple stamps and postal markings, merchants who kept detailed postal reference guides, and international mail that took weeks longer than the physical journey itself required.

The word “Internet” has become so common that it flies under the radar. Take a closer look, and you’ll notice it’s short for “internetwork,” and (as the name suggests) the Internet is not actually a single thing, but a mesh of interconnected networks. Whose networks? Networks owned by companies like Comcast, Verizon, and Tata Communications, but also by companies like Cloudflare, Google, and Netflix. Communication over the Internet is very similar to the postal system: your computer sends a letter to Netflix asking for video, and Netflix sends the video back as a stream of letters. One could imagine a world like mid-19th-century mail, where, to get access to Netflix, you’d need to subscribe to multiple Internet providers. Do you want to watch one show? Sign up for Verizon, Comcast, Cogent, and whatever random transit provider happens to be on the path. But that’s not how the Internet looks from a regular user’s perspective. How come?

The value of a network comes from its size: the larger it is, the more useful it is to join, which in turn makes it more valuable for others to join. And that one sentence is the Internet’s entire personality: a giant network effect wearing a trench coat, pretending to be “just packets,” while doing intense relationship counseling behind the scenes. The slightly awkward truth is that the Internet works not because it is technologically inevitable, but because it is commercially agreeable. The agreements are built out of contracts, norms, grudges, conference-hallway conversations, and the occasional “fine, we’ll plug the cable back in” at 2 a.m. It’s Switzerland-to-Portugal mail, except the horses are networking hardware and the clerks are network engineers with a caffeine dependency.

The Borders of the Internet

If you zoom out, the Internet is a pile of “autonomous systems” (AS-es). An autonomous system is basically a network that can make its own routing decisions. In practice, it often maps to a company (an ISP, a backbone provider, a cloud company, a big content provider), or sometimes a large institution with its own network. Each AS has a number (an ASN), which is like a passport stamp. Each AS has its own “roads” (fiber), its own “post offices” (routers in data centers), and its own economic goals, which mostly involve not doing expensive things for free—unless doing expensive things for free is somehow cheaper than not doing them. Now the key point: these autonomous systems need to exchange traffic. Your Netflix stream is not “a Netflix thing” or “a Comcast thing.” It is a chain of handoffs across multiple autonomous systems. And those handoffs are where the business lives.

There are two broad ways networks exchange traffic. One is transit. Transit is the simplest relationship: “I pay you; you carry my traffic to the rest of the Internet.” You are basically hiring a postal courier who promises to deliver your letters to everyone they can reach. The other is peering. Peering is when two networks connect directly and agree to exchange traffic, often settlement-free (no one pays), though “settlement-free” is sometimes more like “settlement-free until someone gets mad about it.” Peering is the Internet’s version of two neighboring postal administrations saying, “We’ll swap bags of mail at the border. No invoices. Don’t be weird.”

Crucially, peering is voluntary. There is no global law that says Network A must peer with Network B. “Swapping bags of mail at the border” is not one question; it’s twenty questions. Where is the border exchange? One city? Ten cities? Who buys the routers? Who pays for the cross-connect at the data center? How much capacity is there? When it fills up, who upgrades it? What counts as “balanced” traffic? Balanced over what time window—at peak, over a month? Measured how? With what lawyers present? All these considerations highlight that there is a cost to a peering relationship, and whether peering is worth it is a business question.

In early 2014, Ars Technica reported that Netflix’s own speed rankings showed streaming performance on Verizon and Comcast had been dropping for months. This wasn’t a mysterious Wi‑Fi gremlin; it was a widely noted interconnection congestion problem: the “border crossings” between networks were crowded, and the question was whose job it was to widen the bridge. Netflix’s traffic was huge (streaming video tends to do that) and Netflix was often delivering traffic to last-mile ISPs via transit providers like Cogent or Level 3. Comcast customers were pulling a firehose of Netflix bits. Those bits were arriving via someone else’s network, and the interconnect ports between that network and Comcast were congested. At the time, this became a public argument about who should pay. Netflix’s view, very roughly, was that Comcast customers already pay Comcast for Internet; Netflix customers already pay Netflix; therefore, customers already paid everyone; therefore, please connect the networks properly. Comcast’s view, very roughly, was that traffic volumes and imbalance changed; upgrading interconnect capacity isn’t free; maybe the party sending massive traffic should contribute. In February 2014, Comcast and Netflix announced an interconnection agreement meant to provide a better experience for Comcast customers. And then something wonderfully romantic happened: performance improved.

Which brings us neatly to the regulatory question people always ask: “Isn’t this what net neutrality is supposed to stop?” Net neutrality is the Internet’s attempt to hire a referee after discovering that the game is mostly played by people who also own the stadium. The basic consumer intuition is extremely reasonable: “I pay my ISP for Internet access; therefore, I should be able to reach the whole Internet.” And most of the time, you can, which makes the times you can’t feel like fraud. But net neutrality rules have historically focused more directly on what your ISP does inside its own network to your traffic—blocking, throttling, paid prioritization—more than on interconnection agreements between networks. The Netflix-era “paid peering” deals sat awkwardly in the seams of policy. Even though interconnection isn’t exactly the same as last-mile throttling, in human terms, it felt like discrimination. In technical terms, it was a congestion at the borders. In business terms, it was brinkmanship.

Even if net neutrality rules exist, they do not eliminate the underlying economic question of who pays to expand capacity at borders. They can change bargaining power. They can change what kinds of brinkmanship are allowed. They can change what counts as unfair practice. But they cannot make fiber free, or routers immortal, or people less inclined to ask, “Why am I paying for that?”

Intuitively, roughly equal, two-way traffic in a peering relationship seems like a fair deal. But what ratio becomes unfair? A 1:1 ratio feels fair, but what about 1:2, or 1:5, or 1:10? In the Netflix story, the situation was resolved by a friendly handshake. Sometimes, however, a relationship counselor needs to step in—in the form of a national competition regulator.

In 2012, the Autorité de la concurrence (France’s competition authority) issued a press release about a dispute between Cogent (a U.S. transit operator) and France Télécom (Orange), and it is one of those rare documents where a regulator tries to explain Internet plumbing to normal humans. The fight was over whether France Télécom could ask to be paid to open additional interconnection capacity, given that traffic between the networks was highly asymmetric. France Télécom’s policy said it would charge for additional capacity if incoming traffic exceeded outgoing by more than a factor of 2.5. The Autorité said, roughly: 2.5 seems acceptable here. It’s a vibe, put into math.

Cooperation Beats Isolation

You thought you were buying “the Internet,” but you were actually buying access plus a hidden, constantly renegotiated set of treaties that determine whether your data get waved through the border crossing—or told to step aside while the adults argue. The reason the 19th-century postal mess eventually got better wasn’t that horses got faster. It was that bureaucrats standardized agreements and nations decided that making mail reliable was worth mutual compromise. The reason the modern Internet feels like “one Internet” is similar: networks decided, again and again, that interoperability was more profitable than isolation.

But “decided” is doing a lot of work. It’s not one decision; it’s daily posture. It’s peering coordinators sending emails that begin “Per our policy…” and end “Thanks!” It’s engineers noticing graphs at 2 a.m. and politely asking the other side whether they’re also seeing a graph that looks like a ski slope. It’s lawyers reading the words “port congestion” in a court filing and thinking, briefly, that they made a mistake in law school. The Internet is less a machine than a treaty system with fiber optics.

What has brought the Internet together is that everyone wins by working together day-by-day.

Subscribe to the ThousandEyes Blog

Stay connected with blog updates and outage reports delivered while they're still fresh.

Upgrade your browser to view our website properly.

Please download the latest version of Chrome, Firefox or Microsoft Edge.

More detail