lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Pugh <ep...@opensourceconnections.com>
Subject Re: Is solr right for this scenario?
Date Fri, 24 Apr 2009 12:59:25 GMT

On Apr 24, 2009, at 7:54 AM, Developer In London wrote:

> Thanks for the fast reply. Wow this seems a very active community.
>
> I have a few more questions in that case:
>
> 1) If Solr is going to be file-based, os it then preferable to run  
> multiple
> Solrs with Shards? How can I determine what capacity 1 Solr can cope?
It depends!  Solr can manage up to X records easily in a single index,  
however your milage may vary.  One of the nice things about Solr is it  
is very scalable, and offers you many options.   I would go with the  
most simple setup for Solr for now, and then as your development  
progresses, and you load data then investigate sharding etc.  Solr,  
properly managed, won't be your bottleneck, it's be your data loading  
scripts or elsewhere.
>
>
> 2) I am presuming there is already tokenizers for hypertext and xml  
> in Solr
> so that it can use extract the right information out?
There are a number of different options available out there for  
indexing content.
>
>
> 3) I need to also get the 'author' information out for things like  
> blogs. I
> guess theres no universal way of doing it and I have to have someone
> manually go through the documents and feed the solr index with the  
> author
> information?
Your loading script will be bespoke to your situation, however any  
competent developer can put together scripts to load from your varous  
data sources.
>
>
> When you mention 'write a loader script...', do you mean I should
> incorporate the date checking in the loader script? Solr has no  
> internal way
> of checking the timestamp in a document and updating?
Solr makes no assumptions about your data sources, it isn't a document  
management system, it is just a search engine.  Well, that isn't  
totally true, the new DataImportHandler architecture does allow you to  
preserve some information about "when did I last run an update, what  
has been updated since", however it's pretty new stuff.


Eric

>
>
>
>
> Thanks,
>
> Nayeem
>
> 2009/4/24 Eric Pugh <epugh@opensourceconnections.com>
>
>> It seems like you have three components to your system:
>>
>> 1) Data indexing from multiple sources
>>
>> 2) Search for specific words in documents
>>
>> 3) Preserve rating and search term.
>>
>> I think that Solr comes into play on #1 and #2.  You can index  
>> content in
>> any number of approaches, either via the new DataImportHandler  
>> architecture,
>> or the more traditional write a loader script that puts the  
>> documents in
>> Solr.  You can store in Solr when a document was indexed, and use  
>> that to
>> check against the original documents to see if they changed.  Check  
>> a last
>> published tag on an RSS feed, or the last updated time on a  
>> physical file.
>> This is a very common use case for Solr.
>>
>> For #2, you could have users issue queries, and make them  
>> "favorites",
>> storing them in the DB.  Assuming they like the results they mark the
>> documents with the ratings, which you could store in Solr, but I  
>> would put
>> in a DB..  Easier to manage User A says 1, User B says 0.
>>
>> Then for the UI, just issue the search baseed on queries stored in  
>> the db,
>> and match the id's up with the ranking in the DB.  Simple!
>>
>> As far as the last part, Solr works best in filesystem, that is  
>> part of why
>> it is so fast, no clunky SQL.  There are scripts for backing up and
>> restoring indexes that you can use, check the wiki
>> http://wiki.apache.org/solr/SolrOperationsTools.
>>
>> Eric
>>
>>
>>
>>
>> On Apr 24, 2009, at 6:18 AM, Developer In London wrote:
>>
>> Hi All,
>>>
>>> I am new to the whole Solr/Lucene community. But I think this  
>>> might be the
>>> solution ot what I am looking to do. I would appreciate any  
>>> feedback on
>>> how
>>> I can go about doing this with Solr:
>>>
>>> I am looking to make a system where -
>>> a) mainly lots of different blog sites, web journals, articles are  
>>> indexed
>>> on a regular basis. Data that has already been indexed needs to be
>>> revisited
>>> to see if there are any changes.
>>> b) The end users has very fixed search terms, eg 'Lloyds TSB' and
>>> 'Corporate
>>> Banking'. All the documents that are found matching this are  
>>> presented to
>>> a
>>> human to analyse.
>>> c) Once the human analyses the document he gives it a rating of 1,  
>>> 0 or
>>> -1.
>>> This rating needs to be saved somewhere and be linked with the  
>>> specific
>>> document and also with the search term (eg 'Lloyds TSB' & 'Corporate
>>> Banking' in this case).
>>> d) End users can then see these documents with the ratings next to  
>>> them.
>>>
>>> What would be the best approach to this?
>>>
>>> Should I set up a different database to save the rating and relevant
>>> mappings, or is there any way to put it in to Solr?
>>>
>>> My 2nd question is, can Solr Index be saved in a database in any  
>>> way?
>>> Whats
>>> the backup and recovery method on Solr?
>>>
>>> Thanks in advance.
>>>
>>> Nayeem
>>>
>>
>> -----------------------------------------------------
>> Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 |
>> http://www.opensourceconnections.com
>> Free/Busy: http://tinyurl.com/eric-cal
>>
>>
>>
>>
>>
>
>
> -- 
> cashflowclublondon.co.uk
>
>                      ("`-''-/").___..--''"`-._
>                       `6_ 6  )   `-.  (     ).`-.__.`)
>                       (_Y_.)'  ._   )  `._ `. ``-..-'
>                     _..`--'_..-_/  /--'_.' ,'
>                    (il),-''  (li),'  ((!.-'
> .

-----------------------------------------------------
Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com
Free/Busy: http://tinyurl.com/eric-cal





Mime
View raw message