Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: Tobias Hill <Tobias.Hill@citerus.se>
To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>
Date: Thu, 8 Nov 2007 21:28:27 +0100
Subject: SV: OutOfMemory-problems with SortComparatorSource /
 ScoreDocComparator
Thread-Topic: OutOfMemory-problems with SortComparatorSource /
 ScoreDocComparator
Thread-Index: AcgiLbokEVWECphNS8KYvuqkiGHeoQAGDFhM
Message-ID: <BBD1371A36A8D8458F94E4AC71DBEC051588DCA30B@drutten.citerus.net>
References: <BBD1371A36A8D8458F94E4AC71DBEC051588DCA30A@drutten.citerus.net>
In-Reply-To: <BBD1371A36A8D8458F94E4AC71DBEC051588DCA30A@drutten.citerus.net>
Accept-Language: sv-SE, en-US
Content-Language: sv-SE
acceptlanguage: sv-SE, en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Tobias,

I had the similar problem with lucene custom sorting about two years ago.

Please take a look at these two email threads:
http://www.gossamer-threads.com/lists/lucene/java-dev/39100
http://www.gossamer-threads.com/lists/lucene/java-user/38016

It seems like they did not make any patch for this problem since then.
But you could do it yourself - check one of the threads above, there
is small patch that force lucene to not cache in case of Custom
sorting.

P.S. I'm not subscribed (just checking RSS), so you could repost it to
mailing list.

Hope it helps,
Alex


________________________________________
Fr=E5n: Tobias Hill [Tobias.Hill@citerus.se]
Skickat: den 8 november 2007 18:34
Till: java-user@lucene.apache.org
=C4mne: OutOfMemory-problems with SortComparatorSource / ScoreDocComparator

Hi,

We have implemented a custom sort following the pattern in Lucene in Action=
.
Unfortunately this has led to quite serious memory problems. When analyzing
those (with a profiler) it seems that there are as many remaining instances=
 of
our SortComparatorSource as there have been queries against the index.
Moreover, it seems that those are held by only one reference (per instance)
coming from some cache:ish feature of lucene itself.

Caching does not make sense for our comparator ... and if it has to be this
way (for a reason or other) it seems reasonable that the cache dropped the
reference if available memory is getting low so that gc could do it's thing=
. Right?

When digging a bit deeper this also seems to be the intention, since I see
WeakHashMap entries in the reference chain. This leaves me puzzled. Why
are not our instances retained when there is only one incoming reference.
Weak as it seems too.

I would be very thankful if anyone could spot why this custom sorter just
piles up in the cache. See below.

Best regards,
Tobias


/**
 * The BlendSorter can sort on any lucene field and blend its value
 * with the document-score. The blend can be configured.
 */
public class BlendSorter implements SortComparatorSource {

    private final static Log log =3D LogFactory.getLog(BlendSorter.class);
    private FieldConverter fieldConverter;
    private float blendFactor;

    /**
     * Constructs a BlendSorter
     * @param fieldConverter    The converter to use wen converting the luc=
ent field
     * @param blendFactor       The factor to blend (0 means 0% fieldvalue =
and 100%
     *                          document-score, 1 means 100% fieldvalue and=
 0% document-score).
     */
    public BlendSorter(FieldConverter fieldConverter, float blendFactor) {
        this.fieldConverter =3D fieldConverter;
        this.blendFactor =3D blendFactor;
    }

    public ScoreDocComparator newComparator(final IndexReader indexReader, =
final String field) throws IOException {
        return new ScoreDocComparator() {
            Map<Integer, Float> values =3D new HashMap<Integer, Float>();

            public int compare(ScoreDoc scoreDoc1, ScoreDoc scoreDoc2) {
                try {
                    Float v1 =3D getValue(scoreDoc1, indexReader, field);
                    Float v2 =3D getValue(scoreDoc2, indexReader, field);
                    return v2.compareTo(v1);

                } catch (IOException e) {
                    log.error("Cannot read doc", e);
                }
                return 0;
            }

            public Comparable sortValue(ScoreDoc scoreDoc) {
                return values.get(scoreDoc.doc);
            }

            public int sortType() {
                return SortField.FLOAT;
            }


            // lazily get values, and store them in our value-map
            private Float getValue(ScoreDoc scoreDoc, IndexReader indexRead=
er, String field) throws IOException {
                Float value =3D values.get(scoreDoc.doc);
                if (value !=3D null) return value;
                final Document doc =3D indexReader.document(scoreDoc.doc);
                float fieldValue =3D fieldConverter.get(doc.get(field));
                float queryValue =3D scoreDoc.score;
                value =3D blendFactor * fieldValue + (1 - blendFactor) * qu=
eryValue;
                values.put(scoreDoc.doc, value);
                return value;
            }
        };
    }

}


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org