incubator-allura-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Brondsema" <brond...@users.sf.net>
Subject [allura:tickets] #6595 Prevent spiders from requesting tarballs
Date Mon, 26 Aug 2013 22:43:53 GMT
If I do a GET on a rev that has no tarball ever requested, it says "Checking snapshot status..."
and does ajax checks which return 'na' over and over.  We need some way let the user request
the snapshot (put a POST form button right on that page?).

A smaller initial delay is great, but `// Check tarball status every 5 seconds` should be
removed since it's inaccurate now.  The upper limit of 600,000ms seems pretty high too, might
be good to drop that down while you're in there.


---

** [tickets:#6595] Prevent spiders from requesting tarballs**

**Status:** in-progress
**Labels:** stability 
**Created:** Thu Aug 22, 2013 03:43 PM UTC by Dave Brondsema
**Last Updated:** Mon Aug 26, 2013 02:02 PM UTC
**Owner:** Tim Van Steenburgh

The following are examples of spiders requesting tarball creation.  This is unnecessary and
a waste of resources.  We should make it not possible.  We already have `rel=nofollow` but
that apparently isn't working. I think the best solution is to require the URL to be a POST.

~~~~
"GET /p/z-i/code-0/208/tarball HTTP/1.0" 200 16400 "-" "Mozilla/5.0 (compatible; bingbot/2.0;
+http://www.bing.com/bingbot.htm)"
"GET /p/jhotdraw/svn/729/tarball HTTP/1.0" 200 17834 "-" "msnbot/0.01 (+http://search.msn.com/msnbot.htm)"
"GET /p/fourpane/git4pane/ci/ec65df3a5ff2ec7be011c0722286e766c2b76d94/tarball HTTP/1.0" 200
18137 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
"GET /u/lluct/me722-cm/ci/0aa649648a00979ad6ca9e9d61df4e44eb694259/tarball?path=/external/clang
HTTP/1.0" 200 17918 "-" "YisouSpider"
~~~~


---

Sent from sourceforge.net because allura-dev@incubator.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options.
 Or, if this is a mailing list, you can unsubscribe from the mailing list.
Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message