lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: LARM Web Crawler: note on normalized URLs
Date Fri, 21 Jun 2002 13:26:22 GMT
> > It may be even nicer to use some DB implemented in Java, such as
> > HyperSQL (I think that's the name) or Smyle
> > (https://sourceforge.net/projects/smyle/) or Berkeley DB
> > (http://www.sleepycat.com/), although MySQL may be simpler if you
> want
> > to create a crawler that can be run on a cluster of machines that
> share
> > a central link repository.
> 
> Hm, I'll think about it. But MySQL seems to be the KISS way...
> I don't think a central link repository makes sense. Looks like a
> bottleneck to me.

Well, yes, it could become a bottleneck.
However, your crawler is not distributed (yet?), so we don't have to
waste time talking about hypothetical situations.

Otis



__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com

--
To unsubscribe, e-mail:   <mailto:lucene-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-dev-help@jakarta.apache.org>


Mime
View raw message