lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <>
Subject [jira] [Commented] (LUCENE-2793) Directory createOutput and openInput should take an IOContext
Date Fri, 10 Jun 2011 15:04:59 GMT


Simon Willnauer commented on LUCENE-2793:

Varun, your patch doesn't apply cleanly to my latest trunk. i think you should update your
local copy again!

I have trouble to understand how you use MergeInfo etc. I figure there might be a misunderstanding
so here is what I had in mind roughly: 


Index: lucene/src/java/org/apache/lucene/index/
--- lucene/src/java/org/apache/lucene/index/	(revision 1134335)
+++ lucene/src/java/org/apache/lucene/index/	(working copy)
@@ -64,7 +64,7 @@
    *  subset of segments to be merged as well as whether the
    *  new segment should use the compound file format. */
-  public static class OneMerge {
+  public static class OneMerge extends MergeInfo {
     SegmentInfo info;               // used by IndexWriter
     boolean optimize;               // used by IndexWriter
@@ -72,25 +72,26 @@
     long mergeGen;                  // used by IndexWriter
     boolean isExternal;             // used by IndexWriter
     int maxNumSegmentsOptimize;     // used by IndexWriter
-    public long estimatedMergeBytes;       // used by IndexWriter
     List<SegmentReader> readers;        // used by IndexWriter
     List<SegmentReader> readerClones;   // used by IndexWriter
-    public final List<SegmentInfo> segments;
-    public final int totalDocCount;
+    public final List<SegmentInfo> segments = new ArrayList<SegmentInfo>();
     boolean aborted;
     Throwable error;
     boolean paused;
     public OneMerge(List<SegmentInfo> segments) {
+      super(getSegments(segments));
+    }
+    private static int getSegments(List<SegmentInfo> segments) {
       if (0 == segments.size())
         throw new RuntimeException("segments must include at least one segment");
       // clone the list, as the in list may be based off original SegmentInfos and may be
-      this.segments = new ArrayList<SegmentInfo>(segments);
       int count = 0;
       for(SegmentInfo info : segments) {
         count += info.docCount;
-      totalDocCount = count;
+      return count;
     /** Record that an exception occurred while executing

> Directory createOutput and openInput should take an IOContext
> -------------------------------------------------------------
>                 Key: LUCENE-2793
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/store
>            Reporter: Michael McCandless
>            Assignee: Varun Thacker
>              Labels: gsoc2011, lucene-gsoc-11, mentor
>         Attachments: LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch,
LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch
> Today for merging we pass down a larger readBufferSize than for searching because we
get better performance.
> I think we should generalize this to a class (IOContext), which would hold the buffer
size, but then could hold other flags like DIRECT (bypass OS's buffer cache), SEQUENTIAL,
> Then, we can make the DirectIOLinuxDirectory fully usable because we would only use DIRECT/SEQUENTIAL
during merging.
> This will require fixing how IW pools readers, so that a reader opened for merging is
not then used for searching, and vice/versa.  Really, it's only all the open file handles
that need to be different -- we could in theory share del docs, norms, etc, if that were somehow

This message is automatically generated by JIRA.
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message