lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject RE: GData, updateable IndexSearcher
Date Mon, 01 May 2006 18:49:41 GMT
Attached.

It uses subclasses and instanceof which is sort of "hackish" - to do it
correctly requires changes to the base classes.



-----Original Message-----
From: jason rutherglen [mailto:jasonhusong@yahoo.com]
Sent: Monday, May 01, 2006 1:43 PM
To: java-dev@lucene.apache.org
Subject: Re: GData, updateable IndexSearcher


Can you post your code?

----- Original Message ----
From: Robert Engels <rengels@ix.netcom.com>
To: java-dev@lucene.apache.org; jason rutherglen <jasonhusong@yahoo.com>
Sent: Monday, May 1, 2006 11:33:06 AM
Subject: RE: GData, updateable IndexSearcher

fyi, using my reopen(0 implementation (which rereads the deletions)

on a 135mb index, with 5000 iterations

open & close time using new reader = 585609
open & close time using reopen = 27422

Almost 20x faster. Important in a highly interactive/incremental updating
index.

-----Original Message-----
From: jason rutherglen [mailto:jasonhusong@yahoo.com]
Sent: Monday, May 01, 2006 1:24 PM
To: java-dev@lucene.apache.org
Subject: Re: GData, updateable IndexSearcher


I wanted to post a quick hack to see if it is along the correct lines.  A
few of the questions regard whether to resuse existing MultiReaders or
simply strip out only the SegmentReaders.  I do a compare on the segment
name and made it public.  Thanks!


public static IndexReader reopen(IndexReader indexReader) throws IOException
{
    if (indexReader instanceof MultiReader) {
      MultiReader multiReader = (MultiReader)indexReader;

      SegmentInfos segmentInfos = new SegmentInfos();
      segmentInfos.read(indexReader.directory());
      if (segmentInfos.size() == 1) {          // index is optimized
        return SegmentReader.get(segmentInfos, segmentInfos.info(0), false);
      }

      IndexReader[] existingIndexReaders = multiReader.getSubReaders();
      // now go through and compare the segment readers
      Map<String,SegmentReader> existingSegmentMap = new
HashMap<String,SegmentReader>();
      getSegmentReaders(existingIndexReaders, existingSegmentMap);

      Map<String,SegmentInfo> newSegmentInfosMap = new
HashMap<String,SegmentInfo>();

      List<SegmentReader> newSegmentReaders = new
ArrayList<SegmentReader>();

      Iterator segmentInfosIterator = segmentInfos.iterator();
      while (segmentInfosIterator.hasNext()) {
        SegmentInfo segmentInfo = (SegmentInfo)segmentInfosIterator.next();

        if (!existingSegmentMap.containsKey(segmentInfo.name)) {
          // it's new
          SegmentReader newSegmentReader = SegmentReader.get(segmentInfo);
          newSegmentReaders.add(newSegmentReader);
        }
      }
      List<IndexReader> allSegmentReaders = new ArrayList<IndexReader>();
      allSegmentReaders.add(multiReader);
      allSegmentReaders.addAll(newSegmentReaders);

      return new MultiReader(indexReader.directory(), segmentInfos, false,
(IndexReader[])allSegmentReaders.toArray(new IndexReader[0]));
    }
    throw new RuntimeException("indexReader not supported at this time");
  }

  public static void getSegmentReaders(IndexReader[] indexReaders,
Map<String,SegmentReader> map) {
    for (int x=0; x < indexReaders.length; x++) {
      if (indexReaders[x] instanceof MultiReader) {
        MultiReader multiReader = (MultiReader)indexReaders[x];
        IndexReader[] subReaders = multiReader.getSubReaders();
        getSegmentReaders(subReaders, map);
      } else if (indexReaders[x] instanceof SegmentReader) {
        SegmentReader segmentReader = (SegmentReader)indexReaders[x];
        map.put(segmentReader.segment, segmentReader);
      }
    }
  }






Mime
View raw message