httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Reser <>
Subject Re: URL scanning by bots
Date Tue, 30 Apr 2013 23:56:56 GMT
On Tue, Apr 30, 2013 at 4:09 PM, André Warnier <> wrote:
> But I have been trying to figure out a real use case, where expecting 404
> responses in the course of legitimate applications or website access would
> be a normal thing to do, and I admit that I haven't been able to think of
> any.
> Can you come up with an example where this would really be a user case and
> where delying 404 responses would really "break something" ?

Imagine you're a real estate agent.  You have listings for properties,
each one gets a unique URL.  You want search engines to index your
properties that are for sale.  When they sell you want those
properties to stop being shown in search engines.  So you start
returning a 404 for those pages.  For a while you might still show
something about the property with something saying that it's been sold
and some links to other similar properties that aren't sold.
Eventually you purge that property entirely and it's a generic 404
page.  Clearly in such a scenario you don't want to delay 404 pages.

There are of course other examples in other industries.  But basically
any situation where you were you have pages for things that are
temporary you're probably going to want to do something like this.

View raw message