Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "dev@lucene.apache.org" <dev@lucene.apache.org>,
        "yonik@lucidimagination.com" <yonik@lucidimagination.com>
CC: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Date: Fri, 2 Sep 2011 20:53:58 -0700
Subject: Re: Analyzers and sorting with a custom analysis chain
Thread-Topic: Analyzers and sorting with a custom analysis chain
Thread-Index: Acxp7RxbpXvYde8XSV6us7nzzk5jJA==
Message-ID: <52B08674-8CF0-4161-ACA4-A8C4596134BC@jpl.nasa.gov>
References: <AB241937-94E7-44EF-9244-8EC7B48C9E2C@jpl.nasa.gov>
 <CAB_8Yd_3zJ0AzOj80kWwapoK5g_h980=QZw0+0sT0kUJa8bZng@mail.gmail.com>
In-Reply-To: 
 <CAB_8Yd_3zJ0AzOj80kWwapoK5g_h980=QZw0+0sT0kUJa8bZng@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Yonik,

On Sep 2, 2011, at 7:47 PM, Yonik Seeley wrote:

> On Fri, Sep 2, 2011 at 10:26 PM, Mattmann, Chris A (388J)
> <chris.a.mattmann@jpl.nasa.gov> wrote:
>> I'm left with childrenshospitallosangeles as a single token resultant fr=
om the chain.
>> So, when I go to sort the titles in Solr, I use sort=3Dtitle_sort asc, a=
nd I am getting all kinds of weird results when doing
>> a query.
>=20
> Hmmm, a random guess would be that perhaps your analysis chain is
> actually producing more than one token per document.   The lucene
> FieldCache takes the highest for each document (just a non-intended
> side-effect of how the FieldCache entry is populated by enumerating
> terms).
>=20
> Try adding fsv=3Dtrue to your request.  It's an undocumented feature
> used in distributed search (it stands for field sort values) used to
> collate results from different shards.  It should add "sort_values" to
> your response to tell you the sort values for each document.

First off, thanks for the reply. I appreciate it.

I tried the fsv=3Dtrue parameter and it's great, it revealed what's really=
=20
going on here:

 "sort_values":[
  "title_sort",[null,
	null,
	null,
	null,
....

I've got one of those null values for each returned document. Now I guess
I have to find out what's wrong with my CombiningFilter.

All it does basically is have a static method to call incrementToken() and =
then=20
call TermAttribute.term() for each of the tokens in the stream. It takes th=
ese,=20
appends them to a StringBuffer (concats them), and then returns a new=20
KeywordTokenizer providing a StringReader initialized with the merged=20
StringBuffer. Yes, I know this probably isn't the most efficient way and I'=
m=20
open to suggestions.

I think in spelling this out though, I might have elaborated my problem. Si=
nce=20
the method I call in the constructor for my CombiningFilter is super(mergeS=
treamTokens(in))
where mergeStreamTokens is a static method, I think I might have consumed t=
he input=20
TokenStream by the time it gets called for the sort. It works on analysis.j=
sp probably=20
because the stream isn't re-consumed? Not sure, something wiggy is going on=
.

I'll keep poking, thanks again.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org