QUESTIONS?

Updates & insights on all things CDN

Accept-Encoding, It’s Vary important.

Vary: Accept-Encoding

One of the best things about running BootstrapCDN are the new things I've learned about web performance. Today, I'd like to share a few insights gained while resolving an issue originally brought up by a peer. The concern was to include the Vary: Accept-Encoding header -- it's not on every server response, but it should be. Here's why.

When browsers make a request, they include HTTP headers for the server to decide what to send back (Is this a mobile client? Can it handle compressed content? Does it need a certain language?).

That's great for direct access, but modern networks use intermediate caches and CDNs. And there's the problem: how does the cache use headers to decide what to send back? How can it replicate the server's decision-making logic?

Vary to the rescue. The Vary header describes what information "uniquely" identifies a request -- caches should only be used if the incoming request matches the Vary information in the cache.

For example, if a server sends the Vary: User-Agent header, intermediate caches will store a separate cache entry for each User-Agent they see (every OS + browser combination, yikes). This behavior was an issue for me in support (we're hiring!), because we saw origin servers getting hammered as each user-agent requested new content and sidestepped the cache. After some research, I figured out why this happened (turn off Vary: User-Agent), but the header left a bad taste in my mouth.

Well, the BootstrapCDN issue came up, so I decided to give Vary another look. I went to one of my favorite WebPerf education sites and found this article that explains gzip & Accept-Encoding.  After following the communication graphs, I thought "If the Origin Server, CDN and Browser support gzip-encoding, why is an extra header needed?"

There had to be a good reason, so I asked SO.  Two people I respect highly, and had the pleasure of working with at NetDNA (again, we're hiring!), answered the question:

"If for some reason the client has an uncompressed version of the file in its cache, it will know not to subsequently request a compressed version of it again and instead to just use the uncompressed file from the cache." - Kyle Rush 
"Like Kyle said... just replace "client" with a "upstream proxy" (isp, corporate network, etc). So you have the risk of serving uncompressed version to end user that supports gzip, and vice versa." - Sajal Kayan
Imagine two clients: an old browser without compression, and a modern one with it. If they both request the same page, then depending on who sent the request first, the compressed or uncompressed version would be stored in the CDN. Now the problems start: the old browser could ask for a regular "index.html" and get the cached, compressed version (random junk data), or the new browser could get the cached, uncompressed version and try to "unzip" it. Bad news, either way. The fix is for the origin server to send back Vary: Accept-Encoding. Now the intermediate CDNs will keep separate cache entries (one for Accept-encoding: gzip, another if you didn't send the header). These days you're unlikely to have clients without compression, but why risk cache mixups? Origin servers should include Vary: Accept-Encoding, and here's how:

Apache/.htaccess

Nginx

IIS

For those of you reading that are our clients, a pull zone using our CDN will pass the "Vary" header from your origin server - so feel free to set that up! If you don't have good control of your origin server, you can easily add the header using our EdgeRules™ product. Happy Encoding ;)
  • sajalkayan

    Nice post. If the content is “varied” based on any request header, that should be declared in the Vary response header.

    A related post : http://codepolice.net/problems-with-gzip-when-using-iis-7-5-as-an-origin-server-for-a-cdn/

    Thats for setting up IIS correctly to work with few CDNs, but is also relevant for keeping upstream proxies(ISP, corp firewalls) in mind.

  • http://www.facebook.com/marcelo.sulcanieto Marcelo Sulca Nieto

    Gracias por tu ayuda estuve buscando esto
    muchas gracias very good

  • Freddywang

    I think MaxCDN ignores this at the moment. I have js/css resources that return both gzip and non gzipped content depending whether browser send it Accept-Encoding: gzip. But MaxCDN always return me non-gzipped content despite I am using browser that support gzip compression. A huge bummer.

    • http://blog.justindorfman.com jdorfman

      @freddywang:disqus please email me the urls so I can debug. jdorfman at maxcdn dot com

  • http://blog.justindorfman.com jdorfman
  • http://www.rubyops.net/ Joshua Mervine

    Fantastic post! Great info!

Increase Your Website Speed

If You Use Under 15TB/month MaxCDN is for You. It Has:

  • High Performance
  • Easy to Use Control Panel
  • Easy to Setup
  • Low Monthly Fee
LEARN MORE

Recent Posts

Compare the speed of two websites in 5 seconds.

100% Free

Start Testing
TRUSTED BY