Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=dk20050327; d=ix.netcom.com;
  b=WpWrUqjPlrMbiwigf2cqpzKaGwCvhMmBMPbtdSYYtjysfywbFMI1FQWQOeuw2w/H;
  h=Received:Reply-To:From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:X-MSMail-Priority:X-Mailer:In-Reply-To:X-MimeOLE:Importance:X-ELNK-Trace:X-Originating-IP;
Reply-To: <rengels@ix.netcom.com>
From: "Robert Engels" <rengels@ix.netcom.com>
To: <java-dev@lucene.apache.org>,
	"jason rutherglen" <jasonhusong@yahoo.com>
Subject: RE: GData, updateable IndexSearcher
Date: Mon, 1 May 2006 13:33:06 -0500
Message-ID: <LMENLAOACIBLMOIILNNNCEFHFNAA.rengels@ix.netcom.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
In-Reply-To: <20060501182357.25220.qmail@web50008.mail.yahoo.com>
Importance: Normal

fyi, using my reopen(0 implementation (which rereads the deletions)

on a 135mb index, with 5000 iterations

open & close time using new reader = 585609
open & close time using reopen = 27422

Almost 20x faster. Important in a highly interactive/incremental updating
index.

-----Original Message-----
From: jason rutherglen [mailto:jasonhusong@yahoo.com]
Sent: Monday, May 01, 2006 1:24 PM
To: java-dev@lucene.apache.org
Subject: Re: GData, updateable IndexSearcher


I wanted to post a quick hack to see if it is along the correct lines.  A
few of the questions regard whether to resuse existing MultiReaders or
simply strip out only the SegmentReaders.  I do a compare on the segment
name and made it public.  Thanks!


public static IndexReader reopen(IndexReader indexReader) throws IOException
{
    if (indexReader instanceof MultiReader) {
      MultiReader multiReader = (MultiReader)indexReader;

      SegmentInfos segmentInfos = new SegmentInfos();
      segmentInfos.read(indexReader.directory());
      if (segmentInfos.size() == 1) {          // index is optimized
        return SegmentReader.get(segmentInfos, segmentInfos.info(0), false);
      }

      IndexReader[] existingIndexReaders = multiReader.getSubReaders();
      // now go through and compare the segment readers
      Map<String,SegmentReader> existingSegmentMap = new
HashMap<String,SegmentReader>();
      getSegmentReaders(existingIndexReaders, existingSegmentMap);

      Map<String,SegmentInfo> newSegmentInfosMap = new
HashMap<String,SegmentInfo>();

      List<SegmentReader> newSegmentReaders = new
ArrayList<SegmentReader>();

      Iterator segmentInfosIterator = segmentInfos.iterator();
      while (segmentInfosIterator.hasNext()) {
        SegmentInfo segmentInfo = (SegmentInfo)segmentInfosIterator.next();

        if (!existingSegmentMap.containsKey(segmentInfo.name)) {
          // it's new
          SegmentReader newSegmentReader = SegmentReader.get(segmentInfo);
          newSegmentReaders.add(newSegmentReader);
        }
      }
      List<IndexReader> allSegmentReaders = new ArrayList<IndexReader>();
      allSegmentReaders.add(multiReader);
      allSegmentReaders.addAll(newSegmentReaders);

      return new MultiReader(indexReader.directory(), segmentInfos, false,
(IndexReader[])allSegmentReaders.toArray(new IndexReader[0]));
    }
    throw new RuntimeException("indexReader not supported at this time");
  }

  public static void getSegmentReaders(IndexReader[] indexReaders,
Map<String,SegmentReader> map) {
    for (int x=0; x < indexReaders.length; x++) {
      if (indexReaders[x] instanceof MultiReader) {
        MultiReader multiReader = (MultiReader)indexReaders[x];
        IndexReader[] subReaders = multiReader.getSubReaders();
        getSegmentReaders(subReaders, map);
      } else if (indexReaders[x] instanceof SegmentReader) {
        SegmentReader segmentReader = (SegmentReader)indexReaders[x];
        map.put(segmentReader.segment, segmentReader);
      }
    }
  }


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org