Identifying undersea fibre and satellite links with traceroute

I frequently travel to the Middle East, which means I often find myself on the wrong end of a slow Internet link. Sometimes that is oversold undersea fibre, such as in Dubai. More often – because of my work – it is VSAT. I’m an engineer, which means I’m a curious monkey that takes everything apart just to understand how it works. Network topology is one of those things.

I wrote about something similar on my other blog last year.

First, some background information on VSAT. The single biggest limitation of satellite Internet service is latency. In network terms, latency is the total round trip time for your packet to reach its destination and for the reply to return. When we speak of “ping time”, what we mean is latency.

Here are some common latency measurements in milliseconds (ms):

1 ms – within your LAN
25 ms – my home cable service in London to servers located in mainland UK
90 ms – typical home DSL in the US to google.com
100-150 ms – the transatlantic cable between the UK and New York state
600-2000 ms – typical VSAT remote to hub link

Why is VSAT latency so high? Geosynchronous satellites orbit at an altitude of 35,000 km. VSAT remotes connect to the Internet via a star-shaped network with a hub at the centre. When a remote modem pings google.com, its traffic must travel up to the satellite, then back down to our teleport, and then over the terrestrial Internet to google.com. The reply does the opposite, returning back over the terrestrial Internet, back up to the satellite and down again to the remote. That means every single ping you send is travelling 35,000 x 4 = 120,000 km. Divide by c, and you find the minimum latency for any VSAT is 480 ms. Further time is added due to radio modulation, error correction, router delay, and various technical issues with shared bandwidth satellite networks. All that adds up to a minimum effective latency for all VSAT communications of about 600 ms. If that network is busy, it goes up.

So when one of our customers makes a VoIP phone call, there is an audible (600 ms) delay between when he speaks and when he can expect a reply. We constantly work to keep that number as low as possible. People seem to tolerate up to 250 ms without noticing, but when you start edging up to one second, it quickly becomes intolerable – less like a phone, more like a radio.

This latency is immediately noticeable on ping and traceroute. I prefer mtr, a tool that combines the two. Plain old tracert / traceroute will show you the same results, just a lot slower. Here is an example run from a VSAT link:

mtr yahoo.com

 Host                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.5.1                              0.0%    10    1.3   3.3   1.2   7.2   2.4
 2. 172.16.4.97                              0.0%     9  605.8 672.2 605.8 764.5  54.4
 3. 172.16.4.1                               0.0%     9  699.6 690.7 611.9 761.6  50.7
 4. 172.16.3.1                               0.0%     9  637.1 666.6 578.1 763.7  58.6
 5. kln-145-253-9-73.arcor-ip.net            0.0%     9  671.4 638.1 592.0 699.7  38.2
 6. kln-145-254-9-153.arcor-ip.net           0.0%     9  640.2 638.9 578.5 736.1  46.8
 7. dus-145-254-18-194.arcor-ip.net          0.0%     9  655.3 650.9 609.2 684.5  24.2
 8. amd-145-254-16-130.arcor-ip.net          0.0%     9  702.0 667.6 604.8 778.2  60.2
 9. ge-1-3-0.pat1.ams.yahoo.com              0.0%     9  740.9 665.1 595.4 740.9  47.3
10. ge-1-2-0.pat2.ams.yahoo.com              0.0%     8  677.7 673.2 621.4 739.4  37.2
11. so-3-1-0-pat1.the.yahoo.com              0.0%     8  711.9 680.5 616.7 773.4  56.4
12. so-2-1-0-pat1.nyc.yahoo.com              0.0%     8  706.9 706.0 682.0 732.4  17.1
13. so-3-0-0.pat1.dcp.yahoo.com              0.0%     8  685.3 726.9 685.3 755.4  29.3
14. ae2-p170.msr2.re1.yahoo.com              0.0%     8  731.8 747.0 731.3 779.9  18.3
    ae2-p160.msr1.re1.yahoo.com
15. ge-9-3.bas-a1.re4.yahoo.com              0.0%     8  677.9 757.3 677.9 819.1  45.2
    te-8-3.bas-a2.re4.yahoo.com
16. w2.rc.vip.re4.yahoo.com                  0.0%     8  789.1 785.1 708.1 853.4  48.6

Note the huge jump in latency between hops 1 and 2. This must be the space link – hop 1 is the remote modem and hop 2 is the hub modem (the one in the teleport).

The jump in latency is obvious in both directions. From a terrestrial link to a VSAT IP:

 Host                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.1.1                              0.0%    25    0.9   1.1   0.6   6.2   1.1
 2. 192.168.55.34                            0.0%    25    1.1   0.7   0.4   1.1   0.2
 3. 172.16.21.71                             0.0%    25    4.5   4.0   3.6   4.9   0.3
 4. 217.239.41.18                            0.0%    25   39.5  41.5  39.1  67.7   6.9
 5. 193.159.224.254                          0.0%    25   44.6  88.3  44.1 596.8 136.2
 6. 83.137.59.9                              0.0%    24   41.7  42.3  41.4  49.5   1.6
 7. 192.168.140.4                            0.0%    24   42.5  42.7  41.6  44.6   0.6
 8. 192.168.100.50                           0.0%    24   48.1  47.4  46.2  50.3   0.9
 9. 192.168.103.149                          0.0%    24  613.7 719.6 592.2 1128. 127.2
10. 80.86.24.6                               0.0%    24  620.9 787.1 582.9 1678. 272.7

Here the space link is between hops 8 and 9.

Similar increases in latency are detectable along undersea fibre. From my home in London to my alma mater Cal Poly:

 1. gw-belafonte                             0.0%     5    1.0   1.1   1.0   1.2   0.1
 2. 10.181.120.1                             0.0%     5   56.3  41.1  23.5  56.3  14.6
 3. osr01camd-v15.network.virginmedia.net    0.0%     5   32.1  39.9  19.0  89.7  28.8
 4. osr02hari-tenge71.network.virginmedia.n  0.0%     5   40.6  34.6  15.3  54.2  15.0
 5. pop-bb-b-ge-200-0.network.virginmedia.n 20.0%     5   46.1  34.6  18.6  46.1  13.2
 6. gfd-bb-a-so-010-0.network.virginmedia.n 20.0%     5   42.5  47.7  41.7  54.1   6.5
 7. 213.152.245.49                           0.0%     5   23.8  45.8  23.8  65.3  19.1
 8. so-0-0-0.mpr2.ams5.nl.above.net          0.0%     5   46.0  58.0  35.2  83.0  18.3
 9. so-2-0-0.mpr1.lhr2.uk.above.net          0.0%     5   65.2  74.8  47.3 100.2  19.8
10. so-1-1-0.mpr1.dca2.us.above.net          0.0%     5  124.5 120.6 105.2 135.4  10.9
11. so-1-2-0.mpr1.lga5.us.above.net          0.0%     5  132.6 152.9 121.6 216.0  39.1
12. so-2-1-0.mpr1.sjc2.above.net             0.0%     5  187.3 205.8 183.9 251.1  27.0
13. so-4-0-0.mpr3.pao1.us.above.net          0.0%     5  280.9 227.7 200.4 280.9  33.3
14. paix-px1--abovenet.cenic.net            20.0%     5  267.4 232.9 189.2 279.1  46.9
15. dc-svl-core1--svl-dc1-ge.cenic.net       0.0%     5  255.3 240.6 206.3 274.7  26.5
16. dc-sol-agg1--svl-core1-ge.cenic.net      0.0%     5  218.7 216.9 196.6 249.3  19.8
17. dc-sol-agg2--sol-agg1-ge-2.cenic.net    25.0%     5  209.8 209.3 199.1 218.9   9.9
18. dc-slo-dc1--sol-dc2-pos.cenic.net        0.0%     5  240.6 221.6 203.3 240.6  19.9
19. primary-pix-outside.netadm.calpoly.edu  25.0%     5  271.3 247.0 200.7 271.3  40.1
20. 129.65.1.41                              0.0%     5  245.9 213.7 190.7 245.9  23.3
21. ???

Now we can see that my home is named “Belafonte” (hop 1), that I have a network connection via Virgin Media (hops 3-6), that Virgin Media finds it cheaper to route via Above.net through Amsterdam and back to the UK rather than direct within the UK (hops 8 and 9), and that we cross the Atlantic between hops 9 and 10. Also, Cal Poly should know better than to use DNS names that state both function and operating system to anyone looking (hop 19, a Cisco PIX firewall at their network’s border).

Note the 60 ms increase in crossing the Atlantic, which is actually quite good for such a long fibre run. Compare a route from Dubai to the UK:

 1. 192.168.1.1                              0.0%     7    0.5   0.8   0.5   2.4   0.7
 2. 195.229.244.26                           0.0%     7   10.9  18.4   8.0  48.8  14.9
 3. 195.229.245.98                           0.0%     7   15.3  17.0   7.5  28.3   9.4
 4. 194.170.0.234                            0.0%     7    7.5   8.3   7.5   8.8   0.5
 5. 195.229.1.181                            0.0%     7   10.8  14.4  10.8  28.7   6.6
 6. 195.229.1.166                            0.0%     7   14.4  14.8  14.1  15.4   0.5
 7. pos10-0.cr02.hkg05.pccwbtn.net           0.0%     7  292.6 293.7 292.3 298.9   2.6
 8. uunet.pos1-2.cr02.frf02.pccwbtn.net      0.0%     7  258.8 259.1 258.8 259.5   0.3
 9. ge-1-2-0.TL2.FFT1.ALTER.NET              0.0%     6  259.1 259.1 258.6 259.6   0.3
10. so-0-0-0.XR2.LND9.ALTER.NET              0.0%     6  247.2 250.8 246.7 266.3   7.8
11. POS2-0.GW9.LND10.ALTER.NET               0.0%     6  258.3 259.7 258.3 262.4   1.9

Care to spot the fibre there? Here’s the kicker – it looks like it’s routing direct from the UAE to the US, then on to the UK! If memory serves, that’s probably the FLAG cable that runs from the Persian Gulf to the West Coast of the US. Yes, they make runs that long!