lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject RE: Website (crawler for) indexing
Date Wed, 05 Sep 2012 15:14:57 GMT
Please take a look at the Apache Nutch project.  
http://nutch.apache.org/
 
-----Original message-----
> From:Lochschmied, Alexander <Alexander.Lochschmied@vishay.com>
> Sent: Wed 05-Sep-2012 17:09
> To: solr-user@lucene.apache.org
> Subject: Website (crawler for) indexing
> 
> This may be a bit off topic: How do you index an existing website and control the data
going into index?
> 
> We already have Java code to process the HTML (or XHTML) and turn it into a SolrJ Document
(removing tags and other things we do not want in the index). We use SolrJ for indexing.
> So I guess the question is essentially which Java crawler could be useful.
> 
> We used to use wget on command line in our publishing process, but we do no longer want
to do that.
> 
> Thanks,
> Alexander
> 
> 

Mime
View raw message