httpd-modules-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Marantz <>
Subject Vary:User-Agent, best practices, and making the web faster.
Date Sat, 04 Jun 2011 19:26:44 GMT

We've been working on a lingering HTTP-compliance issue in mod_pagespeed:
respecting Vary:User-Agent.  mod_pagespeed needs to cache resources in order
to optimize them.  The economics of this make sense when the server
optimizes a resource, and saves the optimization for serving to multiple

The problem is that this is, in general, expensive to do correctly when the
site owner has put Vary:User-Agent in the response header for, say, a css or
javascript file.  There are legitimate reasons to do this, such as serving a
different version of a CSS file to IE6.  But I think most sites don't do
that.  However, there is a disturbing passage in the document for

SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary

This encourages all site owners to add Vary:User-Agent to all css and js
files, whether they actually vary in content or not.

Does anyone know the history of this recommendation?  Surely that is an
inappropriate recommendation for mod_deflate.  Vary:Accept-Encoding make
sense in the context of that filter, but not Vary:User-Agent.

The problem with defensively setting Vary:User-Agent is that any proxy cache
-- and in this respect mod_pagespeed acts like a proxy cache -- must fetch
origin content for each distinct user-agent.  While it's feasible for us to
employ a level of indirection in our cache so that we only store extra
copies of a resource when it actually differs -- this can, I fear, be
catestrophic to the cache working-set and hit-rate.  We couldn't get around
storing each distinct user-agent.

So, there are two questions:

1. Who can I lobby to get the recommendation changed for the mod_deflate
doc?  That recommendation seems incorrect &/or obsolete.
2. Given that there are likely a huge number of sites that blindly followed
that recommendation, is there a straightforward way for mod_pagespeed to
correct the situation?  Specifically, can mod_pagespeed get access to apache
configuration parameters that were added by other filters, looking
specifically for the pattern quoted above with SetEnvIfNoCace & Header?  Is
it OK for mod_pagespeed to register for config-parameters owned by another

I think what we'd do is basically let mod_pagespeed ignore "Vary:User-Agent"
if we saw that it was inserted per this exact pattern.  This would, to be
pendantic, violate  HTTP, but I think it would help make the web faster, and
in practice would help many more sites than it would hurt.  Sites that
specifically added vary:user-agent using a more specific construct, such as
identifying a particular CSS file that they want to Vary, would be treated


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message