lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From markharw00d <markharw...@yahoo.co.uk>
Subject Re: Searching user-private annotations associated with indexed documents
Date Tue, 27 Nov 2007 22:52:20 GMT
lucene user wrote:
> Am I being clear?
>   

Now you are.
> I don't know what you mean  by "PERSON_ANNOTATION works for Google".
>   
I suppose I meant annotations in the sense GATE and UIMA refer to 
annotations. They are like a highlighter pen marking a particular 
section of a document and adding metadata. It is is possible to 
"overlay" such metadata in the token stream so the sentence "Joe Smith 
works for Google" would have a special "person" annotation token 
introduced *at the same position* as "Joe Smith" and a "company" 
annotation token introduced at the same position as "Google". You then 
have a searchable document that contains both highly specific terms 
("Joe Smith", "Google") and their more generic forms("person", 
"company"). You can then conduct queries to find out which people "work 
for Google" and what companies "Joe Smith works for". Sounds like your 
annotations are a different beast.

One way of weaving your "personalisation" functionality cleanly into 
Lucene would be as follows:
1) Pick up the user's authentication from inside a ServletFilter 
(assuming you have a standard webapp) and set a thread-local "user" 
variable.
2) Overide FilterIndexReader with a new "PersonalisedIndexReader" which 
wraps the standard IndexReader. This class supplies an additional 
"annotations" field in the TermEnum/TermDocs which is personalised 
according to the thread-local "user" variable.

This approach allows you to use all of the standard Lucene functionality 
and yet have a personalised view of the index for each user of the 
system. The join between matches on the personalised "annotations" field 
and the standard Lucene text fields can be optimised because of the 
low-level "TermDocs.skipTo" calls the query logic makes when executing 
Boolean logic. This level of optimisation could not be achieved if you 
did this join outside of Lucene.
The hard bit is managing the details behind storage/retrieval of the 
user's annotations and managing references to doc Ids in this reader but 
overall this seems like a clean way of packaging it all together.

Cheers,
Mark









> On Nov 27, 2007 7:54 AM, mark harwood <markharw00d@yahoo.co.uk> wrote:
>   
>> Do the annotations have positions ?
>>
>> Do you want to do things like phrase-search e.g.
>>      "PERSON_ANNOTATION works for Google"
>>
>> Or is your idea of an annotation more simply a del.ici.ous-style tag associated with
the whole document?
>>
>> Cheers
>> Mark
>>
>>
>> ----- Original Message ----
>> From: lucene user <luz290@gmail.com>
>> To: java-user@lucene.apache.org
>> Sent: Tuesday, 27 November, 2007 12:31:38 PM
>> Subject: Re: Searching user-private annotations associated with indexed documents
>>
>> I'd be VERY grateful for your help, folks! Thanks! I really need some
>> insight on this. THANKS!!
>>
>> On Nov 26, 2007 6:43 PM, lucene user <luz290@gmail.com> wrote:
>>     
>>> Here are the three options that seem practical to us right now.
>>>
>>> (1) Do The annotation search in postgres using LIKE or the
>>>    postgres native full text search. Take the resulting list
>>>    of file ids and use it to build a filter for the lucene query,
>>>    the way we currently do for folders.
>>>
>>> (2) Add a second lucene index that contains only annotations.
>>>    First retrieve a list of file ids satisfying the annotation
>>>    query from this index and use it to create a filter for the
>>>    main lucene query on the archive.
>>>    Whenever annotation text is edited,
>>>      if blank, delete annotation from index
>>>      otherwise add or replace annotation in index.
>>>
>>> (3) Add a second lucene index that contains contentrefs.
>>>     This index would contain the same fields as the arhicve index
>>>     plus the following:
>>>       database_id: list of systemuser_id and contentref_id.
>>>       annotation:  list of all annotation text for this
>>>                    system user and content ref.
>>>       folders:     list of all folder names for this systemuser and
>>>                    content ref
>>>
>>>     Whenever an article is added to or removed from a folder,
>>>     or its annotation text is edited, the following would occur:
>>>       See if it has an entry in the lucene index for the database.
>>>       if so,
>>>         extract the lucene document from the index.
>>>         if the updated list of folders that contain it is empty,
>>>            delete this document from the lucene database index.
>>>         otherwise,
>>>           update the folder and annotation in the document object.
>>>           delete this document from the index.
>>>           add the updated document object to the index.
>>>       if not,
>>>         extract the lucene document for the article from the archive
>>>       
>>  index
>>     
>>>         add the database_id, folders, and annotation fields to this
>>>       
>>  object
>>     
>>>         add the document object to the lucene database index.
>>>
>>> Got a better idea on this?
>>>
>>> Thanks!!
>>>
>>>
>>> On Nov 26, 2007 5:33 PM, lucene user <luz290@gmail.com> wrote:
>>>       
>>>> Folks
>>>>
>>>> I have some additional textual data that is user specific,
>>>>         
>>  basically
>>     
>>>> annotations about documents. I would like to be able to do
>>>> **combined** searches, looking for some words in the document and
>>>>         
>>  some
>>     
>>>> in my users' private annotations about that document. Any
>>>>         
>>  suggestions
>>     
>>>> about how I should handle this? The annotations are changeable by
>>>> users at any time so we have to be ready to delete them and add
>>>>         
>>  others
>>     
>>>> at any time when the user does edit an annotation.
>>>>
>>>> Do I need a second Lucene index? Can I do a query against two
>>>>         
>>  indexes
>>     
>>>> at the same time? If so, how?
>>>>
>>>> The annotations will be very small but highly volatile. The
>>>>         
>>  database
>>     
>>>> of documents will grow large but nothing will be deleted from it.
>>>>
>>>> Thanks!
>>>>
>>>>         
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>>       ___________________________________________________________
>> Yahoo! Answers - Got a question? Someone out there knows the answer. Try it
>> now.
>> http://uk.answers.yahoo.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>   



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message