TL;DR:

  • I was experiencing long load times or outright failures loading some websites using IPv6.
  • My computer’s network interface had its MTU set to 1500.
  • My internet connection uses PPPoE, which supports only 1492 bytes per packet.
  • The packets from the server to my computer were getting dropped.
  • The server advertised a TCP MSS of 1440 (corresponding to an MTU of 1500), and my computer was also advertising TCP MSS of 1440.
  • Therefore, the server sent packets to me of that size, and they were likely dropped by the PPPoE end-point at my ISP, before reaching my modem.
  • Setting the MTU on my network interface to 1492 seems to have resolved the issue, as doing so lowers the MSS negotiated by TCP.
  • Many IPv6 sites worked fine because they advertised a lower MSS.
  • IPv4 worked ok because it can fragment packets on intermediate links.
  • It’s 2017, how can this possibly be something that still needs to be tweaked on a home network?

Background

My internet connection has been acting strangely since I turned on IPv6 last year. The problem was intermittent and only seemed to affect some websites, so it took a while to realize that there was even an issue. I’d been noticing that Wikipedia had been taking a long time to load, or not fully loading pages at all, only to work quickly again on reload.

The last straw came when I found that this website also loaded really slowly. It seemed similar to the problem with Wikipedia, except that it happened much more frequently. That ease of reproducibility made it annoying enough to warrant further investigation, and it made further investigation easier. Also, my personal pride was on the line: my own website should load quickly!

I did have some reason to suspect that IPv6 may be involved as a culprit: another site I host on CloudFront, but with IPv6 disabled, was not experiencing these issues.

System Information

  • ISP is Teksavvy and they do officially support IPv6. If you happen to sign up with TekSavvy because you heard about them through me, please tell them I referred you and give them my customer id: CID328326. I think they’re a great ISP!
  • Networking hardware is a Smart RG 505N modem with integrated NAT/router, switch, and Wi-Fi AP, running firmware 2.5.0.11.
  • Computer is a MacBook Pro running OS X 10.12.3
  • Computer connects via Wi-Fi
  • This website is made up of static files that are stored on S3 and served via CloudFront (see the colophon for more information)

Differentials

Even though I suspected the problem was related to IPv6, there are a lot of far more probable reasons a website might be slow:

  • Web page or assets required by the web page are too large
  • Transfer to my computer is slow
  • Bad caching configuration in CloudFront causing too many misses
  • Scripts on the web page are making it slow

Test 1: Web Request Timeline

The Web Request Timeline shows the problem pretty clearly (horizontal time scale ~10s): Web Request Timeline showing an approximately 5 second request for the HTML page itself, followed by many short, largely parallel requests for the remaining components of the page.

The long request in blue, at the start of the timeline, is for the HTML page itself. The request shows that it takes just under 5 seconds to establish the connection, and then a relatively short period of time for the HTTP request and response.

When I turned IPv6 off, the problem went away (horizontal time scale ~1.5s): Web Request Timeline showing an approximately 500ms request for the HTML page itself, followed by many short, largely parallel requests for the remaining components of the page.

So, the problem has to do with establishing the initial connection, and it is definitely related to IPv6 somehow.

Test 2: Wireshark the IPv6 Connection

The next step is to capture the page load in Wireshark. With IPv6 turned on, the following packet trace was captured: Wireshark packet trace showing errors due to dropped packets from the server during SSL handshake.

There’s a lot going on, so let’s break it down:

Packet Description
199, 202, and 203 Standard 3-way TCP handshake
204 Client begins TLS handshake with a “Client Hello” message
209 Server acknowledges that it received the packet (ACK)
210 Client receives 720 bytes of data from the server, starting at offset 4285.
211 Client uses TCP selective acknowledgment to tell the server that it received those 720 bytes, but it didn’t receive the previous 4284 bytes. (Note that Wireshark erroneously identifies this packet as a duplicate of packet 203.)
… 5 seconds pass! …
267, 268 Server retransmits 1024 bytes of data, starting at offset 1. Client acknowledges.
269, 270 Server retransmits 404 bytes of data, starting at offset 1025. Client acknowledges.
271, 272 Server retransmits 1024 bytes of data, starting at offset 1429. Client acknowledges.
273, 274 Server retransmits 404 bytes of data, starting at offset 2453. Client acknowledges.
275, 276 Server retransmits 1024 bytes of data, starting at offset 2857. Client acknowledges.
277 Server retransmits 404 bytes of data, starting at offset 3881.
278 Client sends a full acknowledgement for all data received since packet 210.
279 Client continues with the next part of the TLS handshake.

So all the packets from 210 to 277, and all that 5+ seconds of time, was spent waiting for the “Server Hello” message to make it to the client. The initial transmission of that message was fast, but the client only received the final 720 bytes… so what happened to the first 4284 bytes?

Analysis of Test 2: Wireshark the IPv6 Connection

It seems like the packets containing those bytes were dropped. One of the reasons that can happen is because the packets are too big. This is the case in IPv4 when the Don’t Fragment flag is set, and is always the case for IPv6, which never fragments a packet on intermediate hops.

Let’s see what would happen if those 4284 bytes were sent as one, two, three, or four, equally-sized packets:

Packets Packet Data Size Liberal MSS Implied MTU
1 4284 4296 4356
2 2142 2154 2214
3 1428 1440 1500
4 1071 1083 1155

MSS is the Maximum Segment Size. It’s a TCP-specific quantity, telling the other end of the connection the maximum number of bytes of data that it can send in a single TCP packet.

The Liberal MSS is computed in this case by adding 12 bytes to the packet data size, since looking at the other TCP segments one see that all of them had 12 bytes of optional TCP headers. Liberal MSS assumes both IP and TCP headers are at their minimum sizes.

MTU is the Maximum Transmission Unit. It’s the maximum size of an IP packet, including IP headers, that can be sent across a data link. Ethernet frames are typically restricted to 1500 bytes of data, which would be the MTU for IP running on Ethernet. If an IP router needs to pass a packet across a data link with a smaller MTU, then the router has to fragment the packet or drop it.

The “Implied MTU” is computed by adding the size of the IP headers and TCP headers to the packet data size. IPv6 headers are 40 bytes, TCP headers are 20 bytes, there are 12 bytes of optional headers, so the value to add is 72.

Note that MTU is typically configured on a network interface, and the OS derives the TCP MSS from the MTU value.

These computations of MSS and MTU are based on the assumption that, when the server has more data to send than will fit in a single TCP segment, it uses packets as large as possible to send the data, finishing with one smaller packet for the remaining data (this is the packet that we actually did receive, packet 210 in the Wireshark trace).

Row 3 of the table really stands out. It shows an MTU of 1500, which is the expected data size for Ethernet. Also, note the MSS of 1440. According to the Wireshark trace, during the TCK handshake, the server responded with a SYN/ACK packet that indicated an MSS of 1440. Knowing these two pieces of information, it’s highly likely that the server tried to send TCP segments containing 1428 bytes of data, and that these packets, taking up 1500 bytes in total, were dropped.

Why are 1500-byte packets getting dropped?

I can’t be sure where they were dropped, but they were probably dropped at my ISP, as it tried to squeeze them into PPPoE to send to my modem. PPPoE embeds 8 bytes of PPP headers inside the data section of an Ethernet frame, leaving 1492 bytes for the IP packet. It’s possible the packets are dropped elsewhere along the way, but we know for sure that, had they made it as far as the other end of my PPPoE connection, they would have been dropped there.

How to Fix the Problem

First, note that the problem does seem to eventually fix itself. The retransmissions in packets 267 to 277 are sent with much smaller packet sizes. And subsequent packets captured by Wireshark also use lower packet sizes. It’s possible that Path MTU Discovery is kicking in. But waiting for the problem to correct itself takes an agonizingly long amount of time. Thankfully, there are several possible fixes for the issue.

Assuming that the packets are being dropped by my PPPoE connection, then fixing it involves somehow telling the remote server to limit IP packet size to 1492 bytes. One way to tell the remote server what to do is given in RFC 879:

On receipt of the MSS option the calculation of the size of segment
that can be sent is:

   SndMaxSegSiz = MIN((MTU - sizeof(TCPHDR) - sizeof(IPHDR)), MSS)

where MSS is the value in the option, and MTU is the Maximum
Transmission Unit (or the maximum packet size) allowed on the
directly attached network.

To paraphrase, the TCP stack will send packets with a maximum data size that is the minimum of a number derived from the MTU of the local interface, and the MSS that is reported by the other end of the connection when the connection is first established.

Currently, my TCP stack is sending an MSS of 1440 when using TCP/IPv6. That can be seen in the Wireshark trace. Since the TCP stack derives MSS from the interface MTU, adjusting the MTU on my local interface should be sufficient:

MacBook-Pro:~ jamesdobson$ networksetup -setMTU en0 1492
MacBook-Pro:~ jamesdobson$

Once this setting takes effect, the Wireshark trace is repeated. Note that it has changed dramatically:

Wireshark packet trace showing successful TCP and SSL handshakes, followed by data exchange.

Note the absence of retransmissions and a quick SSL handshake. Note also that changing the interface MTU locally did, in fact, change the MSS in packet 119: it is now 1432, 60 bytes less than the MTU, leaving enough space for the TCP/IPv6 headers.

The large-sized “Server Hello” packets have 1420 bytes of TCP segment data, 20 bytes of TCP header data, 12 bytes of optional TCP header data, and 40 bytes of IPv6 header data—which adds up to 1492 bytes in total, maxing out what can be transferred across the link.

Problem solved…?

MSS for Some Other Sites

Reducing my MTU made my TCP/IPv6 stack to use an MSS of the correct size, so that packets coming from the server wouldn’t get dropped.

Why wasn’t there a problem with TCP/IPv4 connections to the same server? First, IPv4 allows packet fragmentation on intermediate hops. But, more importantly, the MSS on the server starts at 1452 when using IPv4. Adding 20 bytes for TCP headers and another 20 bytes for IPv4 headers, this gives an MTU of 1492, which is precisely big enough to fit in the PPPoE payload.

One wonders if perhaps CloudFront’s IPv4 stack has had its MSS configured to 1452 to account for consumers accessing it over PPPoE with the wrong MTU configured on their home networks. Perhaps a similar configuration hasn’t yet been applied for IPv6.

The following are the MSS values measured for a small selection of other sites:

Site MSS (IPv4) MSS (IPv6)
www.google.com 1380 1360
pagead46.l.doubleclick.net 1380 1360
scontent.xx.facebook.com 1410 1410
s9.gp1.wac.gammacdn.net 1452 1220
browser-update.org 1452 1220
*.edgecastcdn.net 1452 1220
e4456.dscd.akamaiedge.net 1452 1440
*.cloudfront.net 1452 1440

The Google and Doubleclick values are the most logical to me: the IPv6 MSS should be 20 bytes lower than the IPv4 MSS. A smaller difference, or none at all, seems like a misconfiguration to me.

It’s interesting to note that a few sites use 1220 for the IPv6 MSS, following CloudFlare’s observations of how broken Path MTU Discovery seems to be.

This makes me think that, for any TCP/IP service operated on the public internet, one may be able to get more consistent performance (and perhaps large performance gains for some consumers with mis-configured networks) by setting the MTU and MSS lower than maximum. I plan on making this part of my standard practice going forward.

Appendix A: Diagnostic Tools

Web Request Timeline

Think of this as a Gantt chart showing the requests that a browser makes to load a page. It is integrated into many modern browsers. On Chrome, open the Developer Tools (⌥⌘I), choose the Timeline tab, and press the Record button when ready.

Wireshark

Wireshark captures the packets that pass through a network interface and displays them on screen. It dissects packets of many common network protocols to display them in a meaningful fashion to humans, and to enable the user to craft expressions to filter the data to show just the interesting packets.

Browser Plugin to Show a Site’s Use of IPv4 or IPv6

I am using the Chrome web browser, and there is a decent extension called IPvFoo that displays a page’s IPv6 usage. Just install it directly from the Chrome Web Store from the link provided in the README.md.

Appendix B: Useful Maneuvers

Disable/Enable IPv6 on OS X

To disable IPv6, open a command prompt and type:

MacBook-Pro:~ jamesdobson$ networksetup -setv6off Wi-Fi
MacBook-Pro:~ jamesdobson$

To enable IPv6, use this command:

MacBook-Pro:~ jamesdobson$ networksetup -setv6automatic Wi-Fi
MacBook-Pro:~ jamesdobson$

In both cases, the system will prompt you to enter an administrator password.

Get/Set the MTU of an Interface on OS X

To get the MTU for your network interface, use the following command:

MacBook-Pro:~ jamesdobson$ networksetup -getMTU en0
Active MTU: 1492 (Current Setting: 1492)
MacBook-Pro:~ jamesdobson$

To set the MTU, use this command:

MacBook-Pro:~ jamesdobson$ networksetup -setMTU en0 1492
MacBook-Pro:~ jamesdobson$

When setting MTU, the system will prompt you to enter an administrator password.