lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-2793) Directory createOutput and openInput should take an IOContext
Date Fri, 10 Jun 2011 15:04:59 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047234#comment-13047234
] 

Simon Willnauer commented on LUCENE-2793:
-----------------------------------------

Varun, your patch doesn't apply cleanly to my latest trunk. i think you should update your
local copy again!

I have trouble to understand how you use MergeInfo etc. I figure there might be a misunderstanding
so here is what I had in mind roughly: 

{code}

Index: lucene/src/java/org/apache/lucene/index/MergePolicy.java
===================================================================
--- lucene/src/java/org/apache/lucene/index/MergePolicy.java	(revision 1134335)
+++ lucene/src/java/org/apache/lucene/index/MergePolicy.java	(working copy)
@@ -64,7 +64,7 @@
    *  subset of segments to be merged as well as whether the
    *  new segment should use the compound file format. */
 
-  public static class OneMerge {
+  public static class OneMerge extends MergeInfo {
 
     SegmentInfo info;               // used by IndexWriter
     boolean optimize;               // used by IndexWriter
@@ -72,25 +72,26 @@
     long mergeGen;                  // used by IndexWriter
     boolean isExternal;             // used by IndexWriter
     int maxNumSegmentsOptimize;     // used by IndexWriter
-    public long estimatedMergeBytes;       // used by IndexWriter
     List<SegmentReader> readers;        // used by IndexWriter
     List<SegmentReader> readerClones;   // used by IndexWriter
-    public final List<SegmentInfo> segments;
-    public final int totalDocCount;
+    public final List<SegmentInfo> segments = new ArrayList<SegmentInfo>();
     boolean aborted;
     Throwable error;
     boolean paused;
 
     public OneMerge(List<SegmentInfo> segments) {
+      super(getSegments(segments));
+    }
+
+    private static int getSegments(List<SegmentInfo> segments) {
       if (0 == segments.size())
         throw new RuntimeException("segments must include at least one segment");
       // clone the list, as the in list may be based off original SegmentInfos and may be
modified
-      this.segments = new ArrayList<SegmentInfo>(segments);
       int count = 0;
       for(SegmentInfo info : segments) {
         count += info.docCount;
       }
-      totalDocCount = count;
+      return count;
     }
 
     /** Record that an exception occurred while executing
{code}

> Directory createOutput and openInput should take an IOContext
> -------------------------------------------------------------
>
>                 Key: LUCENE-2793
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2793
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/store
>            Reporter: Michael McCandless
>            Assignee: Varun Thacker
>              Labels: gsoc2011, lucene-gsoc-11, mentor
>         Attachments: LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch,
LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch, LUCENE-2793.patch
>
>
> Today for merging we pass down a larger readBufferSize than for searching because we
get better performance.
> I think we should generalize this to a class (IOContext), which would hold the buffer
size, but then could hold other flags like DIRECT (bypass OS's buffer cache), SEQUENTIAL,
etc.
> Then, we can make the DirectIOLinuxDirectory fully usable because we would only use DIRECT/SEQUENTIAL
during merging.
> This will require fixing how IW pools readers, so that a reader opened for merging is
not then used for searching, and vice/versa.  Really, it's only all the open file handles
that need to be different -- we could in theory share del docs, norms, etc, if that were somehow
possible.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message