Why You Should Peer

The Problem

Two networks that are geographically close may not be directly interconnected and their transit providers networks may not interconnect in many places. Consequently, for data to get from one network to the other it may have to travel 1000's of kilometers and many networks.

Map of 2 ISPs on different backbones

Red Bar Telstra Internet (TI)

Blue Bar connect.com.au (CCA)

Network A is connected to TI and Network B is connected to CCA.

The CCA POP on the Gold Coast is connected to CCA in Brisbane. The Brisbane POP is connected to the CCA POP in Sydney.

The TI POP on the Gold Coast is connected to the TI POP in Sydney.

In Sydney CCA and TI connect to exchange data.

If data has to move from network A to network B then it must travel all the way to Sydney on TI, to Brisbane on CCA and then back to the Gold Coast...possibly metres from where it started.

The Solution

If the two networks peered then the data could go direct to its destination, over the (usually) faster and cheaper peering link.

Example of peering

When data needs to go from Network A to Network B it can now go direct (and for free) over the peering link, bypassing TI and CCA. Network A and Network B will still need to keep their transit (the TI and CCA links) however as they need some way to exchange data with 3rd parties (say Network C).

Some traceroutes to and from various places on the Gold Coast are available. These give some indication of the network topology on the Gold Coast.

The Benefits of Peering

The two main benefits of peering are:

  1. Improved Quality of Service (QoS)
  2. Reduced Costs

1. QoS

As stated, two ISPs on the Gold Coast may be only metres apart but may be far apart in terms of network topology.

No matter how fast everyone's network may be lots of hops and the great distances all add to latency. While this latency might be unnoticeable for web browsing and the like, anything real-time such as telnet, IRC or games will feel it.

This long journey also gives ample opportunity for something to go wrong. The more routers a packet must pass through, the more routers there are to crash. The more cables a packet must pass through, the more cables there are to be dug up by a madman with a back hoe :-)

If the two ISPs peered at a peering point, then traffic would go up one ISP's pipe to the peering point and then down the other ISP's pipe (and vice versa). This would keep the number of hops down, reducing latency and increasing reliability.

ISPs that peer can share resources and even cooperate to obtain new ones and place them at the peering point where everyone has access to them. These can include things like mirror sites, IRC servers and game servers. Having these services locally can greatly improve the service provided to the end user. More on shared services is covered in things that can be done at a peering point.

Another advantage to peering is it gives an ISP a second pipe (if they didn't have one already). This gives redundancy - if the ISP's transit link goes down then they can make a deal with another ISP at the peering point to carry their traffic until their transit link is repaired. Some ISPs have these deals prearranged and if their transit link falls over they automatically switch over to using their neighbour ISP and the customers don't even notice.

2. Reduced Costs

Peering can save an ISP a lot of money. A lot of traffic an ISP carries is geographically local - people who know each other from "real life" who email each other (with their hard drive attached), play Quake against each other and IRC/chat/ICQ with each other. Without peering, all this traffic has to go via the ISP's transit link, being charged at 19c/Mb (or whatever). With a peering point all that traffic goes for free, and doesn't take space up on the ISP's transit link.

Even greater savings occur with proxy peering and news feeds. Peering proxies over the peering point not only saves money but also improves the speed of a user's web browsing. Anyone getting a news feed over their transit link will immediatley see the benefits of sharing news over the peering point, thus spreading the load of the news feed over all ISPs.

Receiving traffic from your peering link, rather than transit link, saves you not just in per byte charges but also reduces the congestion on your transit link. This reduced congestion may allow you to delay costly transit link upgrades without sacrificing performance.

Perhaps the best way to illustrate some of the savings that can be made is to give few figures:

Things To Do At A Peering Point

Excluding telco and equipment costs, it costs no more to have a big, fast pipe to a peering point than a small, slow one. As peering points are usually geographically close to the people peering even telco costs are not that high. This leads to all the ISPs having fast peering links and that opens up a lot of possibilities...

Web Cache Peering

Web cache peering is perhaps the most common activity at an Australian peering point. Its benefits are obvious, both in cost saving and speed. As mentioned above the Victorian Internet Exchange web site cites a 20% reduction in the cost of bandwidth to ISPs who web proxy peer over the peering point.

Mirrors

Mirror sites can be made available to all at the peering point giving everyone fast and free access. Each ISP can host a different mirror or several can get together to share the cost of a mirror. If you have a look at the Squid redir.pl script for WAIX you can see some of the mirrors available via their peering point.

News

Big savings can come from sharing news feeds. I'm unsure how many people today still read USENET but with a local server and some encouragement then the number would probably increase substantially. News can be approached in a number of different ways:

  1. Each ISP could have their own news server. They would receive a news feed from their transit provider and would feed this news onto other ISPs at the peering point. In turn every other ISP would do the same. This way each ISP would have their own news server and a full news feed but only receive a fraction of this feed over their transit link. Unfortunately this would mean each article an ISP received over their transit link would need to be sent over their peering link multiple times (one to each ISPs news server).
  2. A news server could be placed at the peering point. Each ISP would feed it with news they have received over their transit link (and receive a feed from it). This avoids having to send each article multiple times (as occurs in 1) but still gives the benefits of sharing the cost of a full news feed. An ISPs customers would still connect to the ISPs own news server to read the news.
  3. A news server could be placed at the peering point and configured to allow ISPs customers to read from it. Each ISP would still feed the news server but would not need to store news locally. This gives all the benefits of 2 but does not require each ISP to have their own news server.
  4. As 3 but a separate link, just for news could be purchased. This may be a flat rate 64Kb/s ISDN or a satellite feed (I believe there are special rates just for news). Depending on the type of link the peering ISPs may still need to "top up" the news feed from over their transit links.

No matter which approach is taken a goldcoast.* hierarchy could be created for local discussions, the organisation of quake matches, for sale ads and the like. USENET news is a great resource that people often miss out on when ISPs feel maintaining a server is not cost effective. A peering point allows the cost to be shared among many. From an ISPs point of view, the best thing about a news server is that each article comes in once but can be read by many customers.

Game Servers

Peering points are great places to locate game servers. Online games tend to be very susceptible to lag and cause a lot of traffic. A peering point gives a large number of people fast access to the game server and they prevent all the game traffic from having to pass over ISP's transit links. Some peering points, such as the one in Sydney even have IRC servers.

DNS

A caching DNS server can placed at the peering point. A slow DNS can give an ISP a "lagged" feeling, no matter how fast they actually are. Every time a machine tries to make a connection it may have to wait many seconds before it receives a DNS response. A DNS with a large cache at the peering point could benefit all ISPs.

Small ISPs

A more indirect benefit of a peering point comes from the fact that there are a lot of ISPs present at the one point. A small ISP, perhaps just starting up, could connect to the peering point. At the peering point they could purchase transit from one or several of the larger ISPs and, over the one link, they could get access to the resources of the peering point (possibly including a news server and the goldcoast.* hierarchy) and receive transit. By connecting at the peering point the small ISP could choose to buy transit from any number of the other ISPs present without having to have a separate connection to each.

BIX/Ausbone

At some point in the future, if the peering ISPs chose to, a link could be established with the Brisbane Internet Exchange or even Ausbone. This would allow Gold Coast ISPs to peer with those in Brisbane or the rest of Australia.