Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 86737 invoked from network); 20 Oct 2008 12:59:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 20 Oct 2008 12:59:22 -0000 Received: (qmail 91331 invoked by uid 500); 20 Oct 2008 12:59:18 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 91296 invoked by uid 500); 20 Oct 2008 12:59:18 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 91285 invoked by uid 99); 20 Oct 2008 12:59:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Oct 2008 05:59:18 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alexander.aristov@gmail.com designates 74.125.44.28 as permitted sender) Received: from [74.125.44.28] (HELO yx-out-2324.google.com) (74.125.44.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Oct 2008 12:58:10 +0000 Received: by yx-out-2324.google.com with SMTP id 3so259362yxj.5 for ; Mon, 20 Oct 2008 05:58:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=WEUqDnhz0nylPsxdpBjx6/N156g4vlFlTtkE4eWAOTE=; b=DeQfOqcARxEFZV6wJ8u2fURFwBGCDELVotB+MCJoSQFIHIonmhZ+tNS7Ch/NLApH1g Btg+7AvrprauThLfb7JsZvE7Utpv0fjFmoybVY6nazQMUAZifawN9usqT0Bt1+S8T4+2 6FR53h90t6A6mhtKEWxt59br5c8Z6sBZ1lx9w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=dTRiYmQdckjuk79hpNwHytsiDpd1frXvzzeXXBYXK8QVYeP4IkWcgzYArYG/mhvjhk u/pGhLwlJMnAmbURo4NKoC7bw6/X/pwAeQ1xKCmk8aFY2mS2Z1cPDb+eHiXDMrAL2YBb YNq+1XEXEBsWh2A8g2H1jIw75npqvngV9SwJU= Received: by 10.142.154.20 with SMTP id b20mr3044574wfe.99.1224507517745; Mon, 20 Oct 2008 05:58:37 -0700 (PDT) Received: by 10.142.252.20 with HTTP; Mon, 20 Oct 2008 05:58:37 -0700 (PDT) Message-ID: <11975c90810200558m2301021aib391b51ff2428e67@mail.gmail.com> Date: Mon, 20 Oct 2008 16:58:37 +0400 From: "Alexander Aristov" To: java-user@lucene.apache.org Subject: robots.txt MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_102553_3269950.1224507517751" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_102553_3269950.1224507517751 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi all, Just wonder if Nutch takes into consideration rules from the robots.txt file while crawling a site. -- Best Regards Alexander Aristov ------=_Part_102553_3269950.1224507517751--