lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Date Fri, 02 Nov 2007 21:56:15 GMT
Oops.. something else is wrong. I get a "docs out of order" exception  
when building a large index.

Tracking it down...

On Nov 2, 2007, at 4:54 PM, robert engels wrote:

> Should be "... LAST document in a segment." I need to slow down.
>
> On Nov 2, 2007, at 4:49 PM, robert engels wrote:
>
>> Here is the fixed code - need special handling when determining  
>> the length of the document in a segment.
>>
>> FieldsReader.java
>>
>>   final int docs(byte[] buffer,int[] lengths,int start,int max)  
>> throws IOException {
>>       final IndexInput indexStream = (IndexInput) indexLocal.get();
>>       indexStream.seek(start * 8L);
>>       long startoffset = indexStream.readLong();
>>       long lastoffset = startoffset;
>>       int count=0;
>>       while(count<max) {
>> 	  long offset = (start + count + 1 < size) ? indexStream.readLong 
>> () : fieldsStream.length();
>> 	  if(offset-startoffset>buffer.length)
>> 	      break;
>> 	  lengths[count]=(int) (offset-lastoffset);
>> 	  lastoffset = offset;
>> 	  count++;
>>       }
>>       if(count==0)
>> 	  throw new IllegalStateException("buffer too small for a single  
>> document");
>> 	
>>       final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
>>       fieldsStream.seek(startoffset);
>>       fieldsStream.readBytes(buffer,0,(int) (lastoffset- 
>> startoffset));
>>
>>       return count;
>>   }
>>
>>
>> On Nov 2, 2007, at 4:33 PM, robert engels wrote:
>>
>>> Oops. Be careful with that code, merging works but indexing  
>>> fails... I have a bug lurking...
>>>
>>> On Nov 2, 2007, at 4:28 PM, robert engels wrote:
>>>
>>>> As expected it did not make much difference. Here is the code:
>>>>
>>>> FieldsReader.java
>>>>   final int docs(byte[] buffer,int[] lengths,int start,int max)  
>>>> throws IOException {
>>>>       final IndexInput indexStream = (IndexInput) indexLocal.get();
>>>>       indexStream.seek(start * 8L);
>>>>       long startoffset = indexStream.readLong();
>>>>       long lastoffset = startoffset;
>>>>       int count=0;
>>>>       while(count<max) {
>>>> 	  long offset = indexStream.readLong();
>>>> 	  if(offset-startoffset>buffer.length)
>>>> 	      break;
>>>> 	  lengths[count]=(int) (offset-lastoffset);
>>>> 	  lastoffset = offset;
>>>> 	  count++;
>>>>       }
>>>>       if(count==0)
>>>> 	  throw new IllegalStateException("buffer too small for a  
>>>> single document");
>>>> 	
>>>>       final IndexInput fieldsStream = (IndexInput)  
>>>> fieldsLocal.get();
>>>>       fieldsStream.seek(startoffset);
>>>>       fieldsStream.readBytes(buffer,0,(int) (lastoffset- 
>>>> startoffset));
>>>>
>>>>       return count;
>>>>   }
>>>>
>>>> FieldsWriter.java
>>>>     final void addDocuments(byte[] buffer, int[] lengths, int  
>>>> ndocs) throws IOException {
>>>> 	long position = fieldsStream.getFilePointer();
>>>> 	long start = position;
>>>> 	for(int i=0;i<ndocs;i++) {
>>>> 	    indexStream.writeLong(position);
>>>> 	    position+=lengths[i];
>>>> 	}
>>>>         fieldsStream.writeBytes(buffer,(int) (position-start));
>>>>     }
>>>>
>>>> SegmentReader.java
>>>>   public int documents(byte[] buffer,int[] lengths,int start,int  
>>>> max) throws IOException {
>>>>       return fieldsReader.docs(buffer,lengths,start,max);
>>>>   }
>>>>
>>>> SegmentMerger.java
>>>>   private final int mergeFields() throws IOException {
>>>> 	fieldInfos = new FieldInfos(); // merge field names
>>>> 	int docCount = 0;
>>>> 	for (int i = 0; i < readers.size(); i++) {
>>>> 	    IndexReader reader = (IndexReader) readers.elementAt(i);
>>>> 	    if (reader instanceof SegmentReader) {
>>>> 		SegmentReader sreader = (SegmentReader) reader;
>>>> 		for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
>>>> 		    FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
>>>> 		    fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,  
>>>> fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, ! 
>>>> reader.hasNorms(fi.name));
>>>> 		}
>>>> 	    } else {
>>>> 		addIndexed(reader, fieldInfos, reader.getFieldNames 
>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET), true,  
>>>> true, true);
>>>> 		addIndexed(reader, fieldInfos, reader.getFieldNames 
>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true,  
>>>> false);
>>>> 		addIndexed(reader, fieldInfos, reader.getFieldNames 
>>>> (IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false,  
>>>> true);
>>>> 		addIndexed(reader, fieldInfos, reader.getFieldNames 
>>>> (IndexReader.FieldOption.TERMVECTOR), true, false, false);
>>>> 		addIndexed(reader, fieldInfos, reader.getFieldNames 
>>>> (IndexReader.FieldOption.INDEXED), false, false, false);
>>>> 		fieldInfos.add(reader.getFieldNames 
>>>> (IndexReader.FieldOption.UNINDEXED), false);
>>>> 	    }
>>>> 	}
>>>> 	fieldInfos.write(directory, segment + ".fnm");
>>>>
>>>> 	SegmentReader[] sreaders = new SegmentReader[readers.size()];
>>>> 	for (int i = 0; i < readers.size(); i++) {
>>>> 	    IndexReader reader = (IndexReader) readers.elementAt(i);
>>>> 	    boolean same = reader.getFieldNames().size() ==  
>>>> fieldInfos.size() && reader instanceof SegmentReader;
>>>> 	    if(same) {
>>>> 		SegmentReader sreader = (SegmentReader) reader;
>>>> 		for (int j = 0; same && j < fieldInfos.size(); j++) {
>>>> 		    same = fieldInfos.fieldName(j).equals(sreader.getFieldInfos 
>>>> ().fieldName(j));
>>>> 		}
>>>> 		if(same)
>>>> 		    sreaders[i] = sreader;
>>>> 	    }
>>>> 	}
>>>> 	
>>>> 	byte[] buffer = new byte[512*1024];
>>>> 	int[] lengths = new int[10000];
>>>>
>>>> 	// merge field values
>>>> 	FieldsWriter fieldsWriter = new FieldsWriter(directory,  
>>>> segment, fieldInfos);
>>>> 	
>>>> 	try {
>>>> 	    for (int i = 0; i < readers.size(); i++) {
>>>> 		IndexReader reader = (IndexReader) readers.elementAt(i);
>>>> 		SegmentReader sreader = sreaders[i];
>>>> 		int maxDoc = reader.maxDoc();
>>>> 		for (int j = 0; j < maxDoc;)
>>>> 		    if (!reader.isDeleted(j)) { // skip deleted docs
>>>> 			if (sreader!=null) {
>>>> 			    int start=j;
>>>> 			    int ndocs=1;
>>>> 			    for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+

>>>> +,ndocs++);
>>>> 			    ndocs = sreader.documents(buffer,lengths,start,j);
>>>> 			    fieldsWriter.addDocuments(buffer,lengths,ndocs);
>>>> 			} else {
>>>> 			    fieldsWriter.addDocument(reader.document(j));
>>>> 			    j++;
>>>> 			}
>>>> 			docCount++;
>>>> 		    }
>>>> 	    }
>>>> 	} finally {
>>>> 	    fieldsWriter.close();
>>>> 	}
>>>> 	return docCount;
>>>>     }
>>>>
>>>>
>>>>
>>>>
>>>> On Nov 2, 2007, at 3:49 PM, robert engels wrote:
>>>>
>>>>> I am working on a bulk document copy right now - I will let you  
>>>>> know if it improves things much.
>>>>>
>>>>> I doubt it, because I already configure the streams to use  
>>>>> fairly large input and output buffers during a merge - but the  
>>>>> memory index merge may see additional benefits due to lower CPU  
>>>>> calls.
>>>>>
>>>>>
>>>>> On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
>>>>>
>>>>>>
>>>>>>     [ https://issues.apache.org/jira/browse/LUCENE-1043? 
>>>>>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment- 
>>>>>> tabpanel#action_12539696 ]
>>>>>>
>>>>>> robert engels commented on LUCENE-1043:
>>>>>> ---------------------------------------
>>>>>>
>>>>>> When bulk copying the documents, I think you need to:
>>>>>>
>>>>>> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]  
>>>>>> offsets;
>>>>>> calculate length = offset[ndocs]-offset[0];
>>>>>> read bytes of length from document file
>>>>>> startoffset = current output document stream position
>>>>>> write bytes to output document
>>>>>> modify offset[] adding startoffset-offset[0] to each entry
>>>>>> write offset[] in bulk to index output
>>>>>>
>>>>>>> Speedup merging of stored fields when field mapping "matches"
>>>>>>> -------------------------------------------------------------
>>>>>>>
>>>>>>>                 Key: LUCENE-1043
>>>>>>>                 URL: https://issues.apache.org/jira/browse/ 
>>>>>>> LUCENE-1043
>>>>>>>             Project: Lucene - Java
>>>>>>>          Issue Type: Improvement
>>>>>>>          Components: Index
>>>>>>>    Affects Versions: 2.2
>>>>>>>            Reporter: Michael McCandless
>>>>>>>            Assignee: Michael McCandless
>>>>>>>            Priority: Minor
>>>>>>>             Fix For: 2.3
>>>>>>>
>>>>>>>         Attachments: LUCENE-1043.patch
>>>>>>>
>>>>>>>
>>>>>>> Robert Engels suggested the following idea, here:
>>>>>>>   http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>>>>>>> When merging in the stored fields from a segment, if the  
>>>>>>> field name ->
>>>>>>> number mapping is identical then we can simply bulk copy the
 
>>>>>>> entire
>>>>>>> entry for the document rather than re-interpreting and then 

>>>>>>> re-writing
>>>>>>> the actual stored fields.
>>>>>>> I've pulled the code from the above thread and got it working
 
>>>>>>> on the
>>>>>>> current trunk.
>>>>>>
>>>>>> -- 
>>>>>> This message is automatically generated by JIRA.
>>>>>> -
>>>>>> You can reply to this email to add a comment to the issue online.
>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------

>>>>>> ----
>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------ 
>>>>> ---
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>>>>
>>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message