lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darin McBeath <ddmcbe...@yahoo.com.INVALID>
Subject Re: SpanQuery not working as expected
Date Fri, 06 Jun 2014 17:03:00 GMT
Thanks Tim.

I have thought about this for the author field (and like you suggest) it would probably work.
 I was actually going to experiment with this later today.

But, I have another field that has a bit more nesting (and it contains authors)

For example, within a given document, I have the following:

References [ one or more]
  Authors [one or more]
     First Name
     Last Name

 
So, I would need to search for a specific author (matching first name and last name) within
a specific reference for a document.  With this double level of nesting, I don't think the
multivalued field approach would work (please correct me if I'm wrong).  That's why I decided
to use span queries.  My index has more than 100 fields, but I only have 2 or 3 fields that
require this structure search capability.  There are also many documents (100M) so I didn't
really want to get into a parent-child type approach.

Plus, there are also many other fields (both within an author) and within an individual reference
that need to be scoped.  For example, there is a 'source tittle' at the reference level and
an 'article title' at the reference level. I would need to search within a given reference
where the 'source title' contains some words, where the 'article title' contains some words,
and within this reference where a specific author contains 'john' for the first name and 'smith'
in the last name.

I guess I'm curious if what I was doing with the SpanQuery should have worked, whether I misunderstood
something, or if this is a bug.

Darin.




________________________________
From: "Allison, Timothy B." <tallison@mitre.org>
To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>; Darin McBeath <ddmcbeath@yahoo.com>

Sent: Friday, June 6, 2014 10:12 AM
Subject: RE: SpanQuery not working as expected


Hi Darin,

Have you thought about using multivalued fields?  If you set the positionIncrementGap to
something kind of big (well > 1, say :) ), and you know that your data is always authorfirst,
authorlast,  you could just search for "darin fulford".

The positionincrementgap will prevent matching on doc2 below.

Doc1
Authorsfield:
    Darin fulford

Doc2 
Authorsfield:
    Matilda darin
    Fulford alexandria

Don't get me wrong, I love the capabilities of SpanQuery, but will this simple solution meet
your needs?





-----Original Message-----
From: Darin McBeath [mailto:ddmcbeath@yahoo.com.INVALID] 
Sent: Thursday, June 05, 2014 7:17 PM
To: java-user@lucene.apache.org
Subject: SpanQuery not working as expected

I read through the http://searchhub.org/2009/07/18/the-spanquery/ which provided a good
overview for how one can construct fairly complex span queries.  I was particularly interested
in the ability to construct nested span queries.  I'm trying to apply this concept to search
a field that contains some structure (as below).  I have a couple of other fields that will
have a bit more nesting, but this should give the general idea.  

authors
  author [one or more]
    first name
    last name

Prior to indexing the content with Lucene, I added some 'markers' around the various bits
I might want to search.  For example 'bauthor' implies beginning author, 'eauthor' implies
ending author, and 'sauthor' implies a separator between individual authors (that would be
used as part of the exclude clause in a not span query).  I do similar things for 'first
name' and 'last name'.

My constructed query (as interpreted by Lucene) is included below.  This was extracted from
the 'parsed string' returned from the query when I set debug=true.  Within a given 'authscope'
field, I'm trying to find a situation where the author first name is 'darin' and the last
name is 'fulford' within a given 'author'.   

spanNot(
    spanNear(
        [authscope:bauthor, 
        spanNear(
            [spanNot(
                spanNear(
                    [authscope:bfname, 
                    authscope:darin, 
                    authscope:efname], 
                    2147483647, true), 
                authscope:sfname, 0, 0), 
             spanNot(
                spanNear(
                    [authscope:blname, 
                    authscope:fulford, 
                    authscope:elname], 
                    2147483647, true), 
                authscope:slname, 0, 0)], 
             2147483647, false), 
         authscope:eauthor], 
         2147483647, true), 
     authscope:sauthor, 0, 0)",

I have loaded the following  2 documents into my index.

[
  {"id":"1", "authscope":" bauthors  bauthor blname mcbeath elname slname  bfname 
darin efname sfname  eauthor sauthor  bauthor blname  fulford elname slname  bfname 
darby efname sfname  eauthor sauthor  bauthor blname  mcbeath elname slname  bfname 
darby efname sfname  eauthor sauthor  eauthors sauthors "},
  {"id":"2", "authscope":" bauthors  bauthor blname  mcbeath elname slname  bfname 
darin efname sfname  eauthor sauthor  bauthor blname  fulford elname slname  bfname 
darin efname sfname  eauthor sauthor  eauthors sauthors "}
]

What I can't figure out is why the above query would match on both documents.  It should
only match the document with id:2.


Any insights would be appreciated.  I'm using Lucene 4.7.2.

Darin.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message