lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manfred Schäfer <>
Subject Re: Proposal for Lucene
Date Fri, 08 Feb 2002 10:26:49 GMT

i would suggest two sub-projects:

1.Crawler - retrieving docs, wherever they are.....

2. DocumentHandler extract Text, create apropriate fields etc..

The second is a layer on top of lucene. First is a autonomous package, wich
should be nicely integrated with lucene/Document-Handler, but should also be
usable for other projects.

I've included my code, to show you, what i've done. It isn't too useful yet,
because it is integrated in our product, but you can get the idea. Actually i've
written two things:

1: A robot for crawling a remote server via http and writing all the data to
local filesystem, then importing it into our db and
(at the same time) replacing all links with internal links. So we could emulate
a web-Site from this crawled Data!

2: (I've rewritten some of the code from 1 for that, so this is much cleaner) A
customer needs a tool for importing local mini-Websites on the file-system via
an applet, send it to the Web-Server and import it as described in point 1. I've
tried to write it in a way, that it could include the functionality of point 1
(retrieving vie http), but that is mostly untested.

I don't say, that you(we) should use this. But i think it's time to come to a
more concrete plans. I'm interested to help on that for the crawler.



View raw message