From users-return-11658-apmail-jackrabbit-users-archive=jackrabbit.apache.org@jackrabbit.apache.org Thu Jun 18 15:31:15 2009 Return-Path: Delivered-To: apmail-jackrabbit-users-archive@minotaur.apache.org Received: (qmail 57318 invoked from network); 18 Jun 2009 15:31:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 18 Jun 2009 15:31:14 -0000 Received: (qmail 810 invoked by uid 500); 18 Jun 2009 15:31:25 -0000 Delivered-To: apmail-jackrabbit-users-archive@jackrabbit.apache.org Received: (qmail 781 invoked by uid 500); 18 Jun 2009 15:31:24 -0000 Mailing-List: contact users-help@jackrabbit.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@jackrabbit.apache.org Delivered-To: mailing list users@jackrabbit.apache.org Received: (qmail 770 invoked by uid 99); 18 Jun 2009 15:31:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2009 15:31:24 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [64.18.2.161] (HELO exprod7og104.obsmtp.com) (64.18.2.161) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 18 Jun 2009 15:31:14 +0000 Received: from source ([209.85.220.205]) by exprod7ob104.postini.com ([64.18.6.12]) with SMTP ID DSNKSjpdrLr3UagzhkA/W0Jjy1JTtTMf22Gt@postini.com; Thu, 18 Jun 2009 08:30:53 PDT Received: by mail-fx0-f205.google.com with SMTP id 1so1287858fxm.41 for ; Thu, 18 Jun 2009 08:30:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.105.195 with SMTP id u3mr1355797fao.13.1245339052305; Thu, 18 Jun 2009 08:30:52 -0700 (PDT) In-Reply-To: <697f8380906180513s7cd3ee19vf2f9751a8ea676a7@mail.gmail.com> References: <9B333D29-9580-414C-B451-8CAB9EBA6ED1@tfd.co.uk> <697f8380906170204t5e90030bp215fbadaeee1386d@mail.gmail.com> <9BC2FD1A-9A05-4717-9F5B-E85A5B827027@tfd.co.uk> <697f8380906171433o22d8e8fbi3992b3fcac26f2ad@mail.gmail.com> <8014BEDB-9C42-405F-8AC4-4EFE67CE147C@tfd.co.uk> <697f8380906180037w441f2338tb5a59c7889c16b98@mail.gmail.com> <697f8380906180513s7cd3ee19vf2f9751a8ea676a7@mail.gmail.com> Date: Thu, 18 Jun 2009 17:30:52 +0200 Message-ID: <697f8380906180830r5a963052w5079f4db41c73a1d@mail.gmail.com> Subject: Re: Query that sorts a large result set. From: Ard Schrijvers To: users@jackrabbit.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org > I have been having these thoughts as well, did not implement anything, > but already expected the 50% drop: I namely think, that you dropped > the SharedFieldCache memory usage with for example 90%, where the > SharedFieldCache is a little more then the actual used memory: namely, > the other 50% is kept by the opened indexReader internal in lucene. > *If* we are able to actively flush this (TermEnum cache) in our opened > lucene indexes, I think we are there: namely the SharedFieldCache > contains the sorting data already, which is kept. Then, we might drop > memory to way beyond 50%. I looked at this quite some time ago, but > did not have time to see how we could flush the lucene termenums... Small addon: >From lucene api: /lucene/docs/api/org/apache/lucene/index/TermEnum.html close() Closes the enumeration to further activity, freeing resources. As the SharedFieldCache does end in the finally with termEnum.close(); one would expect Lucene to free the in memory terms....I did not test it, but from the top of my head, memory snapshots I saw in the past do indicate memory is not freed, and the terms are kept in memory Regards Ard > >