Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=wAjZk/hKZ2ir+522nE7CEm27jWCEhI3uQc9kQ/OVZRyirgFXsHXWWUifX8UNV1Hv8zFaTQJd0SBJaqrOfdwlbjqMux0kJHaNenZ4YBHsAnjgkBSTX5N5v7wFUst4egdZr1S4LKmF2yWhkpapMvACshvQGhsGPganxkip834jBr8=;
Message-ID: <43867.57165.qm@web52902.mail.re2.yahoo.com>
Date: Fri, 15 Jan 2010 23:26:26 -0800 (PST)
From: Ahmet Arslan <iorixxx@yahoo.com>
Subject: Re: A way to download URLs and index better ?
To: java-user@lucene.apache.org
In-Reply-To: <3b23ce091001152320k661a52bcr2a1381cf49eecc67@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

> Hi everyone, please help me this
> question:
> I need downloading some webpages from a list of URLs (about
> 200 links) and
> then index them by Lucene.
> This list is not fixed, because it depends on definition of
> my process.
> Currently, in my web application, I wrote class for
> downloading, but it
> download time is too long.
> 
> Please recommend me a Java library suitable with my
> situation for optimize
> downloading.
> More its examples are very wonderful (INPUT: list of URLs;
> OUTPUT: webpages
> content, or indexed repository)
> Thank you very much.

Probably most famous ones :

http://lucene.apache.org/nutch/
http://crawler.archive.org/


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org