lucene user wrote:
> Am I being clear?
>
Now you are.
> I don't know what you mean by "PERSON_ANNOTATION works for Google".
>
I suppose I meant annotations in the sense GATE and UIMA refer to
annotations. They are like a highlighter pen marking a particular
section of a document and adding metadata. It is is possible to
"overlay" such metadata in the token stream so the sentence "Joe Smith
works for Google" would have a special "person" annotation token
introduced *at the same position* as "Joe Smith" and a "company"
annotation token introduced at the same position as "Google". You then
have a searchable document that contains both highly specific terms
("Joe Smith", "Google") and their more generic forms("person",
"company"). You can then conduct queries to find out which people "work
for Google" and what companies "Joe Smith works for". Sounds like your
annotations are a different beast.
One way of weaving your "personalisation" functionality cleanly into
Lucene would be as follows:
1) Pick up the user's authentication from inside a ServletFilter
(assuming you have a standard webapp) and set a thread-local "user"
variable.
2) Overide FilterIndexReader with a new "PersonalisedIndexReader" which
wraps the standard IndexReader. This class supplies an additional
"annotations" field in the TermEnum/TermDocs which is personalised
according to the thread-local "user" variable.
This approach allows you to use all of the standard Lucene functionality
and yet have a personalised view of the index for each user of the
system. The join between matches on the personalised "annotations" field
and the standard Lucene text fields can be optimised because of the
low-level "TermDocs.skipTo" calls the query logic makes when executing
Boolean logic. This level of optimisation could not be achieved if you
did this join outside of Lucene.
The hard bit is managing the details behind storage/retrieval of the
user's annotations and managing references to doc Ids in this reader but
overall this seems like a clean way of packaging it all together.
Cheers,
Mark
> On Nov 27, 2007 7:54 AM, mark harwood <markharw00d@yahoo.co.uk> wrote:
>
>> Do the annotations have positions ?
>>
>> Do you want to do things like phrase-search e.g.
>> "PERSON_ANNOTATION works for Google"
>>
>> Or is your idea of an annotation more simply a del.ici.ous-style tag associated with
the whole document?
>>
>> Cheers
>> Mark
>>
>>
>> ----- Original Message ----
>> From: lucene user <luz290@gmail.com>
>> To: java-user@lucene.apache.org
>> Sent: Tuesday, 27 November, 2007 12:31:38 PM
>> Subject: Re: Searching user-private annotations associated with indexed documents
>>
>> I'd be VERY grateful for your help, folks! Thanks! I really need some
>> insight on this. THANKS!!
>>
>> On Nov 26, 2007 6:43 PM, lucene user <luz290@gmail.com> wrote:
>>
>>> Here are the three options that seem practical to us right now.
>>>
>>> (1) Do The annotation search in postgres using LIKE or the
>>> postgres native full text search. Take the resulting list
>>> of file ids and use it to build a filter for the lucene query,
>>> the way we currently do for folders.
>>>
>>> (2) Add a second lucene index that contains only annotations.
>>> First retrieve a list of file ids satisfying the annotation
>>> query from this index and use it to create a filter for the
>>> main lucene query on the archive.
>>> Whenever annotation text is edited,
>>> if blank, delete annotation from index
>>> otherwise add or replace annotation in index.
>>>
>>> (3) Add a second lucene index that contains contentrefs.
>>> This index would contain the same fields as the arhicve index
>>> plus the following:
>>> database_id: list of systemuser_id and contentref_id.
>>> annotation: list of all annotation text for this
>>> system user and content ref.
>>> folders: list of all folder names for this systemuser and
>>> content ref
>>>
>>> Whenever an article is added to or removed from a folder,
>>> or its annotation text is edited, the following would occur:
>>> See if it has an entry in the lucene index for the database.
>>> if so,
>>> extract the lucene document from the index.
>>> if the updated list of folders that contain it is empty,
>>> delete this document from the lucene database index.
>>> otherwise,
>>> update the folder and annotation in the document object.
>>> delete this document from the index.
>>> add the updated document object to the index.
>>> if not,
>>> extract the lucene document for the article from the archive
>>>
>> index
>>
>>> add the database_id, folders, and annotation fields to this
>>>
>> object
>>
>>> add the document object to the lucene database index.
>>>
>>> Got a better idea on this?
>>>
>>> Thanks!!
>>>
>>>
>>> On Nov 26, 2007 5:33 PM, lucene user <luz290@gmail.com> wrote:
>>>
>>>> Folks
>>>>
>>>> I have some additional textual data that is user specific,
>>>>
>> basically
>>
>>>> annotations about documents. I would like to be able to do
>>>> **combined** searches, looking for some words in the document and
>>>>
>> some
>>
>>>> in my users' private annotations about that document. Any
>>>>
>> suggestions
>>
>>>> about how I should handle this? The annotations are changeable by
>>>> users at any time so we have to be ready to delete them and add
>>>>
>> others
>>
>>>> at any time when the user does edit an annotation.
>>>>
>>>> Do I need a second Lucene index? Can I do a query against two
>>>>
>> indexes
>>
>>>> at the same time? If so, how?
>>>>
>>>> The annotations will be very small but highly volatile. The
>>>>
>> database
>>
>>>> of documents will grow large but nothing will be deleted from it.
>>>>
>>>> Thanks!
>>>>
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>> ___________________________________________________________
>> Yahoo! Answers - Got a question? Someone out there knows the answer. Try it
>> now.
>> http://uk.answers.yahoo.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
|