incubator-allura-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Brondsema" <>
Subject [allura:tickets] #6595 Prevent spiders from requesting tarballs
Date Thu, 22 Aug 2013 22:31:47 GMT
I wonder if we'd want to be a bit more nuanced.  If I click on "Download Snapshot" and then
am waiting for the zip to be generated, I might hit refresh, and then I get a page that says
"405 Method Not Allowed".  Would it be practical to have a GET request check for status only?
 If there were no snapshot ready or in-progress, we'd probably need a message & link to
POST a new request.

That would also allow people to share URLs (e.g. in an email or webpage) directly to the code
snapshot page still.


** [tickets:#6595] Prevent spiders from requesting tarballs**

**Status:** code-review
**Labels:** stability 
**Created:** Thu Aug 22, 2013 03:43 PM UTC by Dave Brondsema
**Last Updated:** Thu Aug 22, 2013 10:00 PM UTC
**Owner:** Tim Van Steenburgh

The following are examples of spiders requesting tarball creation.  This is unnecessary and
a waste of resources.  We should make it not possible.  We already have `rel=nofollow` but
that apparently isn't working. I think the best solution is to require the URL to be a POST.

"GET /p/z-i/code-0/208/tarball HTTP/1.0" 200 16400 "-" "Mozilla/5.0 (compatible; bingbot/2.0;
"GET /p/jhotdraw/svn/729/tarball HTTP/1.0" 200 17834 "-" "msnbot/0.01 (+"
"GET /p/fourpane/git4pane/ci/ec65df3a5ff2ec7be011c0722286e766c2b76d94/tarball HTTP/1.0" 200
18137 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3;"
"GET /u/lluct/me722-cm/ci/0aa649648a00979ad6ca9e9d61df4e44eb694259/tarball?path=/external/clang
HTTP/1.0" 200 17918 "-" "YisouSpider"


Sent from because is subscribed to

To unsubscribe from further messages, a project admin can change settings at
 Or, if this is a mailing list, you can unsubscribe from the mailing list.
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message