httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier>
Subject Re: URL scanning by bots
Date Wed, 01 May 2013 00:23:11 GMT
Ben Reser wrote:
> On Tue, Apr 30, 2013 at 4:09 PM, André Warnier <> wrote:
>> But I have been trying to figure out a real use case, where expecting 404
>> responses in the course of legitimate applications or website access would
>> be a normal thing to do, and I admit that I haven't been able to think of
>> any.
>> Can you come up with an example where this would really be a user case and
>> where delying 404 responses would really "break something" ?
> Imagine you're a real estate agent.  You have listings for properties,
> each one gets a unique URL.  You want search engines to index your
> properties that are for sale.  When they sell you want those
> properties to stop being shown in search engines.  So you start
> returning a 404 for those pages.  For a while you might still show
> something about the property with something saying that it's been sold
> and some links to other similar properties that aren't sold.
> Eventually you purge that property entirely and it's a generic 404
> page.  Clearly in such a scenario you don't want to delay 404 pages.
> There are of course other examples in other industries.  But basically
> any situation where you were you have pages for things that are
> temporary you're probably going to want to do something like this.

Thank you.  That is a good example.

Alternatives :
1) if you were running such a site (which I would still suppose is a minority of the 600 
Million websites which exist), you could easily disable the feature.
2) you could instead return a redirect response, to a page saying "that one was sold, but

look at these".
That may be even more friendly to search engines, and to customers.

View raw message