Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of williamstw@gmail.com
 designates 209.85.198.230 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=qJZnjCzxjLHXw7QF622opIEUhq1YziBpSRG9UvCxAoqYKmiKGGcOY+Fvl33SSWklZ/
         mcSlBj45z0yGOR7txahEhjyMy/E4RtWSj7m6+HDdeGyyOgCjBGjgloIOhOkWeIvRaC1U
         1shDhDL2/U1TgrTcqBbUbHeEGrZDWH3vXoHfU=
MIME-Version: 1.0
In-Reply-To: <0BA554DD-9D7D-4AC1-B8A0-A4026EE38538@apache.org>
References: <2c6101c99d10$620df810$0301a8c0@tmesa.com>
	 <0BA554DD-9D7D-4AC1-B8A0-A4026EE38538@apache.org>
Date: Wed, 4 Mar 2009 19:02:05 -0500
Message-ID: <499888440903041602g5f658274l24978d67cd35b95d@mail.gmail.com>
Subject: Re: crawler questions..
From: Tim Williams <williamstw@gmail.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Mar 4, 2009 at 4:41 PM, Grant Ingersoll <gsingers@apache.org> wrote=
:
> You might have a look at Droids (http://incubator.apache.org/droids/) or
> Nutch (http://lucene.apache.org/nutch) and their communities. =A0They are=
 much
> more focused on crawling (not to say there aren't people here who crawl,
> just saying those projects are (mostly) about crawling)
>
>
> On Mar 4, 2009, at 4:30 PM, bruce wrote:
>
>> Hi...
>>
>> Sorry that this is a bit off track. Ok, maybe way off track!
>>
>> But I don't have anyone to bounce this off of..
>>
>> I'm working on a crawling project, crawling a college website, to extrac=
t
>> course/class information. I've built a quick test app in python to crawl
>> the
>> site. I crawl at the top level, and work my way down to getting the
>> required
>> course/class schedule. The app works. I can consistently run it and
>> extract
>> the information.
>>
>> My issue is now that I have a "basic" app that works, i need to figure o=
ut
>> how I guarantee that I'm correctly crawling the site. How do I know when
>> I've got an error at a given node/branch, so that the app knows that it'=
s
>> not going to fetch the underlying branch/nodes of the tree..
>>
>> How do I know when I have a complete "tree"!
>>
>> I'm looking for someone, or some group/prof that I can talk to about the=
se
>> issues. My goal is to eventually look at using nutch/lucene if at all
>> applicable.
>>
>> Any pointers, or people, or papers, etc... would be helpful.

The ir-book[1] also has a section on the subject if you hadn't already
seen it.

--tim

[1] - http://nlp.stanford.edu/IR-book/html/htmledition/web-crawling-and-ind=
exes-1.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org