lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Burlison <Alan.Burli...@sun.com>
Subject Re: Handling disparate data sources in Solr
Date Thu, 04 Jan 2007 21:53:12 GMT
Original problem statement:

----------
I'm considering using Solr to replace an existing bare-metal Lucene 
deployment - the current Lucene setup is embedded inside an existing 
monolithic webapp, and I want to factor out the search functionality 
into a separate webapp so it can be reused more easily.

At present the content of the Lucene index comes from many different 
sources (web pages, documents, blog posts etc) and can be different 
formats (plaintext, HTML, PDF etc).  All the various content types are 
rendered to plaintext before being inserted into the Lucene index.

The net result is that the data in one field in the index (say 
"content") may have come from one of a number of source document types. 
  I'm having difficulty understanding how I might map this functionality 
onto Solr.  I understand how (for example) I could use 
HTMLStripStandardTokenizer to insert the contents of a HTML document 
into a field called "content", but (assuming I'd written a PDF analyser) 
how would I insert the content of a PDF document into the same "content" 
field?

I know I could do this by preprocessing the various document types to 
plaintext in the various Solr clients before inserting the data into the 
index, but that means that each client would need to know how to do the 
document transformation.  As well as centralising the index, I also want 
to centralise the handling of the different document types.
----------

My initial suggestion, to get the discussion started, is to extend the 
<doc> and <field> element with the following attributes:

mime-type
Mime type of the document, e.g. application/pdf, text/html and so on.

encoding
Encoding of the document, with base64 being the standard implementation.

href
The URL of any documents that can be accessed over HTTP, instead of 
embedding them in the indexing request.  The indexer would fetch the 
document using the specified URL.

There would then be entries in the configuration file that map each MIME 
type to a handler that is capable of dealing with that document type.

Thoughts?

-- 
Alan Burlison
--

Mime
View raw message