Return-Path: X-Original-To: apmail-incubator-allura-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-allura-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4FD951084B for ; Mon, 26 Aug 2013 22:44:31 +0000 (UTC) Received: (qmail 98104 invoked by uid 500); 26 Aug 2013 22:44:31 -0000 Delivered-To: apmail-incubator-allura-dev-archive@incubator.apache.org Received: (qmail 98077 invoked by uid 500); 26 Aug 2013 22:44:31 -0000 Mailing-List: contact allura-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: allura-dev@incubator.apache.org Delivered-To: mailing list allura-dev@incubator.apache.org Received: (qmail 98059 invoked by uid 99); 26 Aug 2013 22:44:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Aug 2013 22:44:19 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of noreply@sourceforge.net designates 216.34.181.60 as permitted sender) Received: from [216.34.181.60] (HELO smtp.ch3.sourceforge.com) (216.34.181.60) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Aug 2013 22:44:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.com; s=x; h=Date:Message-ID:Subject:Reply-To:From:To:MIME-Version:Content-Type; bh=GiLvQdfEhK900DDVkqXrXDYRf/FzGU5aV/DsLUWi9zI=; b=XH7R+r7fa+FW2Bwx3SlZmVkttWnjqNwXeblGrgquo7QOzAhB/28qznAaoBYyRskAuZuN0/yNcUhrWiTAs8+bqPVulmPc9P7ij4Hwd45qCWCdu1sZUsAgLkQR6sqmW8rxAp0bv3KKxbVfb4a3/kSe8o2TbhMuIT3nEldGEDtv2R8=; Received: from localhost ([127.0.0.1] helo=sfs-alluradaemon-4.v29.ch3.sourceforge.com) by sfs-alluradaemon-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1VE5Vp-00045U-U4 for allura-dev@incubator.apache.org; Mon, 26 Aug 2013 22:43:53 +0000 Content-Type: multipart/related; boundary="===============3955428832199183237==" MIME-Version: 1.0 To: "[allura:tickets] " <6595@tickets.allura.p.re.sf.net> From: "Dave Brondsema" Reply-To: "[allura:tickets] " <6595@tickets.allura.p.re.sf.net> Subject: [allura:tickets] #6595 Prevent spiders from requesting tarballs Message-ID:

Date: Mon, 26 Aug 2013 22:43:53 +0000 X-Virus-Checked: Checked by ClamAV on apache.org --===============3955428832199183237== Content-Type: multipart/alternative; boundary="===============7077499506660072356==" MIME-Version: 1.0 --===============7077499506660072356== MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit - **status**: code-review --> in-progress --- ** [tickets:#6595] Prevent spiders from requesting tarballs** **Status:** in-progress **Labels:** stability **Created:** Thu Aug 22, 2013 03:43 PM UTC by Dave Brondsema **Last Updated:** Mon Aug 26, 2013 02:02 PM UTC **Owner:** Tim Van Steenburgh The following are examples of spiders requesting tarball creation. This is unnecessary and a waste of resources. We should make it not possible. We already have `rel=nofollow` but that apparently isn't working. I think the best solution is to require the URL to be a POST. ~~~~ "GET /p/z-i/code-0/208/tarball HTTP/1.0" 200 16400 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "GET /p/jhotdraw/svn/729/tarball HTTP/1.0" 200 17834 "-" "msnbot/0.01 (+http://search.msn.com/msnbot.htm)" "GET /p/fourpane/git4pane/ci/ec65df3a5ff2ec7be011c0722286e766c2b76d94/tarball HTTP/1.0" 200 18137 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)" "GET /u/lluct/me722-cm/ci/0aa649648a00979ad6ca9e9d61df4e44eb694259/tarball?path=/external/clang HTTP/1.0" 200 17918 "-" "YisouSpider" ~~~~ --- Sent from sourceforge.net because allura-dev@incubator.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/ To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list. --===============7077499506660072356== MIME-Version: 1.0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: 7bit
  • status: code-review --> in-progress

[tickets:#6595] Prevent spiders from requesting tarballs

Status: in-progress
Labels: stability
Created: Thu Aug 22, 2013 03:43 PM UTC by Dave Brondsema
Last Updated: Mon Aug 26, 2013 02:02 PM UTC
Owner: Tim Van Steenburgh

The following are examples of spiders requesting tarball creation. This is unnecessary and a waste of resources. We should make it not possible. We already have rel=nofollow but that apparently isn't working. I think the best solution is to require the URL to be a POST.

"GET /p/z-i/code-0/208/tarball HTTP/1.0" 200 16400 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
"GET /p/jhotdraw/svn/729/tarball HTTP/1.0" 200 17834 "-" "msnbot/0.01 (+http://search.msn.com/msnbot.htm)"
"GET /p/fourpane/git4pane/ci/ec65df3a5ff2ec7be011c0722286e766c2b76d94/tarball HTTP/1.0" 200 18137 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
"GET /u/lluct/me722-cm/ci/0aa649648a00979ad6ca9e9d61df4e44eb694259/tarball?path=/external/clang HTTP/1.0" 200 17918 "-" "YisouSpider"

Sent from sourceforge.net because allura-dev@incubator.apache.org is subscribed to https://sourceforge.net/p/allura/tickets/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/allura/admin/tickets/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.

--===============7077499506660072356==-- --===============3955428832199183237==--