Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: <37531382449432@webcorp1h.yandex-team.ru>
References: <13891381357052@webcorp1g.yandex-team.ru>
 <CALr4Hzpyg51h5PHELVYQZ8BLQZwE4XGhxtXozqzqx2uU5fzO4A@mail.gmail.com>
 <40781382021812@webcorp1h.yandex-team.ru>
 <CAL8PwkbMDHh-Ok93PFWfDUZDyi2cxFH0FkPQGSbba8QHBP0Kvw@mail.gmail.com>
 <34621382047527@webcorp1g.yandex-team.ru>
 <CAL8PwkZXXKNrGDEnf8+0+d1JzZMuRDfsgSbEhL2XngNDgnJoFw@mail.gmail.com>
 <107051382133024@webcorp1g.yandex-team.ru>
 <CAL8PwkYK++unTKo+DV_yUhiaDqPTs2r=Johp_w6LEkh7uU7teg@mail.gmail.com>
 <37531382449432@webcorp1h.yandex-team.ru>
From: Michael McCandless <lucene@mikemccandless.com>
Date: Wed, 23 Oct 2013 11:05:32 -0400
Message-ID: 
 <CAL8PwkZ1mM3uTqUtS961_6Bernq+LkNrN2nk+C-e0ubu+V4Q-A@mail.gmail.com>
Subject: Re: Lucene in-memory index
To: Lucene Users <java-user@lucene.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Oct 22, 2013 at 9:43 AM, Igor Shalyminov
<ishalyminov@yandex-team.ru> wrote:

> Thanks for the link, I'll definitely dig into SpanQuery internals very so=
on.

You could also just make a custom query.  If you start from the
ProxBooleanTermQuery on that issue, but change it so that it rejects
hits that didn't have terms in the right positions, then you'll likely
have a much faster way to do your query.

>>>  For "A,sg" and "A,pl" I use unordered SpanNearQueries with the slop=3D=
-1.
>>
>> I didn't even realize you could pass negative slop to span queries.
>> What does that do?  Or did you mean slop=3D1?
>
> I indeed use an unordered SpanNearQuery with the slop =3D --1 (I saw it o=
n some forum, maybe here: http://www.gossamer-threads.com/lists/lucene/java=
-user/89377?do=3Dpost_view_flat#89377)

Wow, OK.  I have no idea what slop=3D-1 does...

> So far it works for me:)
>
>>
>>>  I wrap them into an ordered SpanNearQuery with the slop=3D0.
>>>
>>>  I see getPayload() in the profiler top. I think I can emulate payload =
checking with cleverly assigned position increments (and then maximum posit=
ion in a document might jump up to ~10^9 - I hope it won't blow the whole i=
ndex up).
>>>
>>>  If I remove payload matching and keep only position checking, will it =
speed up everything, or the positions and payloads are the same?
>>
>> I think it would help to avoid payloads, but I'm not sure by how much.
>>  E.g., I see that NearSpansOrdered creates a new Set for every hit
>> just to hold payloads, even if payloads are not going to be used.
>> Really the span scorers should check Terms.hasPayloads up front ...
>>
>>>  My main goal is getting the precise results for a query, so proximity =
boosting won't help, unfortunately.
>>
>> OK.
>>
>> I wonder if you can somehow identify the spans you care about at
>> indexing time, e.g. A,sg followed by N,sg and e.g. add a span into the
>> index at that point; this would make searching much faster (it becomes
>> a TermQuery).  For exact matching (slop=3D0) you can also index
>> shingles.
>
> Thanks for the clue, I think it can be a good optimization heuristic.
> I actually tried a similar approach to optimize search of attributes at t=
he same position.
> Here's how it was supposed to work for a feature set "S,sg,nom,fem":
>
> * the regular approach: split it into grammar atomics: "S", "sg", "nom", =
"fem". With payloads and positions assigned the right way, this would allow=
 us to search for an arbitrary combination of these attributes _but_ with m=
ultiple postings merging.
> * the experimental approach: sort the atomics lexicographically and index=
 all the subsets: "S", "fem", "nom", "sg", "S,fem", "S,nom", ..., "S,fem,no=
m,sg". With the preprocessing of the user query the same way (split - sort =
- join) it would allow us to process the same queries exactly within one po=
sting.
>
> This technique is actually used in our current production index based on =
Yandex.Server engine.
> But Yandex.Server somehow makes the index size reasonable (within the ord=
er of magnitude of original text size), while Lucene index blows up totally=
 ( >10 times original text size) and no search performance improvements app=
ear.

That's really odd.  I would expect index to become much larger, but
search performance ought to be much faster since you run simple
TermQuery.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org