lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: strategy for post-processing answer set
Date Thu, 29 Sep 2011 01:08:47 GMT

: it looks to me as if Solr just brings back the URLs. what I want to do is to
: get the actual documents in the answer set, simplify their HTML and remove
: all the javascript, ads, etc., and append them into a single document.
: 
: Now ... does Nutch already have the documents? can I get them from its db?
: or do I have to go get the documents again with something like a wget?

i *think* what you are saying is that:

a) you built your index using nutch
b) when you query Solr, you only get back a "url" field for each matching 
document 
c) what you want is to combine the whole text of webpages corrisponding to 
all of those urls into one massive html page

If that's the case,then you should either:

1) ask on the nutch-user mailing list about how to "store" the whole 
content of web pages that nutch crawls so you cna build up a page like 
this (nutch may already be doing it, i don't know -- depends on the 
schema)

2) write custom client code (probably outside of the score of velocity) to 
re-fetch these urls "at query time" and parse them and combine them as you 
see fit.

which approach is right for you all depends on your goals and use case -- 
but solr can only give you back the fields you store in it.

-Hoss

Mime
View raw message