lucene-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sim...@apache.org
Subject svn commit: r1160229 - in /lucene/dev/branches/branch_3x/lucene: ./ backwards/ backwards/src/test/org/apache/lucene/index/ contrib/misc/src/java/org/apache/lucene/store/ contrib/misc/src/test/org/apache/lucene/index/ contrib/misc/src/test/org/apache/lu...
Date Mon, 22 Aug 2011 12:07:32 GMT
Author: simonw
Date: Mon Aug 22 12:07:31 2011
New Revision: 1160229

URL: http://svn.apache.org/viewvc?rev=1160229&view=rev
Log:
LUCENE-3218: back out changes due to issues with delegating directories

Added:
    lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestCompoundFile.java
      - copied unchanged from r1138440, lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestCompoundFile.java
    lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestIndexFileDeleter.java
      - copied unchanged from r1138440, lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestIndexFileDeleter.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/CompoundFileReader.java
      - copied unchanged from r1138440, lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/CompoundFileReader.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/CompoundFileWriter.java
      - copied unchanged from r1138440, lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/CompoundFileWriter.java
Removed:
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/CompoundFileDirectory.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/CompoundFileWriter.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/DefaultCompoundFileDirectory.java
    lucene/dev/branches/branch_3x/lucene/src/test-framework/org/apache/lucene/store/MockCompoundFileDirectoryWrapper.java
Modified:
    lucene/dev/branches/branch_3x/lucene/   (props changed)
    lucene/dev/branches/branch_3x/lucene/CHANGES.txt
    lucene/dev/branches/branch_3x/lucene/backwards/   (props changed)
    lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestAddIndexes.java
    lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
    lucene/dev/branches/branch_3x/lucene/contrib/misc/src/java/org/apache/lucene/store/NRTCachingDirectory.java
    lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestIndexSplitter.java
    lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/store/TestNRTCachingDirectory.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/DocumentsWriter.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexFileNames.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexReader.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexWriter.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentCoreReaders.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfo.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfos.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentMerger.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/DataOutput.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/Directory.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/FileSwitchDirectory.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/IndexOutput.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/MMapDirectory.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/NIOFSDirectory.java
    lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/SimpleFSDirectory.java
    lucene/dev/branches/branch_3x/lucene/src/test-framework/org/apache/lucene/store/MockDirectoryWrapper.java
    lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestAddIndexes.java
    lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
    lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestCompoundFile.java
    lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java
    lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/store/TestFileSwitchDirectory.java

Modified: lucene/dev/branches/branch_3x/lucene/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/CHANGES.txt?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/CHANGES.txt (original)
+++ lucene/dev/branches/branch_3x/lucene/CHANGES.txt Mon Aug 22 12:07:31 2011
@@ -75,13 +75,6 @@ New Features
 
 Optimizations
   
-* LUCENE-3201, LUCENE-3218: CompoundFileSystem code has been consolidated 
-  into a Directory implementation. Reading is optimized for MMapDirectory,
-  NIOFSDirectory and SimpleFSDirectory to only map requested parts of the
-  CFS into an IndexInput. Writing to a CFS now tries to append to the CF
-  directly if possible and merges separately written files on the fly instead
-  of during close. (Simon Willnauer, Robert Muir)
-
 * LUCENE-3289: When building an FST you can now tune how aggressively
   the FST should try to share common suffixes.  Typically you can
   greatly reduce RAM required during building, and CPU consumed, at

Modified: lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestAddIndexes.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestAddIndexes.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestAddIndexes.java (original)
+++ lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestAddIndexes.java Mon Aug 22 12:07:31 2011
@@ -949,7 +949,7 @@ public class TestAddIndexes extends Luce
     w3.addIndexes(readers);
     w3.close();
     
-    assertEquals("Only one compound segment should exist", 4, dir.listAll().length);
+    assertEquals("Only one compound segment should exist", 3, dir.listAll().length);
   }
  
   // LUCENE-2996: tests that addIndexes(IndexReader) applies existing deletes correctly.

Modified: lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java (original)
+++ lucene/dev/branches/branch_3x/lucene/backwards/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java Mon Aug 22 12:07:31 2011
@@ -520,6 +520,87 @@ public class TestBackwardsCompatibility 
     return indexDir;
   }
 
+  /* Verifies that the expected file names were produced */
+
+  public void testExactFileNames() throws IOException {
+
+    String outputDirName = "lucene.backwardscompat0.index";
+    File outputDir = _TestUtil.getTempDir(outputDirName);
+    _TestUtil.rmDir(outputDir);
+
+    try {
+      Directory dir = newFSDirectory(outputDir);
+
+      LogMergePolicy mergePolicy = newLogMergePolicy(true, 10);
+      mergePolicy.setNoCFSRatio(1); // This test expects all of its segments to be in CFS
+      IndexWriterConfig conf = newIndexWriterConfig(TEST_VERSION_CURRENT, new WhitespaceAnalyzer(TEST_VERSION_CURRENT)).setMaxBufferedDocs(-1).setRAMBufferSizeMB(16.0)
+        .setMergePolicy(mergePolicy);
+      IndexWriter writer = new IndexWriter(dir, conf);
+      for(int i=0;i<35;i++) {
+        addDoc(writer, i);
+      }
+      assertEquals("wrong doc count", 35, writer.maxDoc());
+      writer.close();
+
+      // Delete one doc so we get a .del file:
+      IndexReader reader = IndexReader.open(dir, false);
+      Term searchTerm = new Term("id", "7");
+      int delCount = reader.deleteDocuments(searchTerm);
+      assertEquals("didn't delete the right number of documents", 1, delCount);
+
+      // Set one norm so we get a .s0 file:
+      reader.setNorm(21, "content", (float) 1.5);
+      reader.close();
+
+      // The numbering of fields can vary depending on which
+      // JRE is in use.  On some JREs we see content bound to
+      // field 0; on others, field 1.  So, here we have to
+      // figure out which field number corresponds to
+      // "content", and then set our expected file names below
+      // accordingly:
+      CompoundFileReader cfsReader = new CompoundFileReader(dir, "_0.cfs");
+      FieldInfos fieldInfos = new FieldInfos(cfsReader, "_0.fnm");
+      int contentFieldIndex = -1;
+      for(int i=0;i<fieldInfos.size();i++) {
+        FieldInfo fi = fieldInfos.fieldInfo(i);
+        if (fi.name.equals("content")) {
+          contentFieldIndex = i;
+          break;
+        }
+      }
+      cfsReader.close();
+      assertTrue("could not locate the 'content' field number in the _2.cfs segment", contentFieldIndex != -1);
+
+      // Now verify file names:
+      String[] expected = new String[] {"_0.cfs",
+                               "_0_1.del",
+                               "_0_1.s" + contentFieldIndex,
+                               "segments_2",
+                               "segments.gen"};
+
+      String[] actual = dir.listAll();
+      Arrays.sort(expected);
+      Arrays.sort(actual);
+      if (!Arrays.equals(expected, actual)) {
+        fail("incorrect filenames in index: expected:\n    " + asString(expected) + "\n  actual:\n    " + asString(actual));
+      }
+      dir.close();
+    } finally {
+      _TestUtil.rmDir(outputDir);
+    }
+  }
+
+  private String asString(String[] l) {
+    String s = "";
+    for(int i=0;i<l.length;i++) {
+      if (i > 0) {
+        s += "\n    ";
+      }
+      s += l[i];
+    }
+    return s;
+  }
+
   private void addDoc(IndexWriter writer, int id) throws IOException
   {
     Document doc = new Document();

Modified: lucene/dev/branches/branch_3x/lucene/contrib/misc/src/java/org/apache/lucene/store/NRTCachingDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/contrib/misc/src/java/org/apache/lucene/store/NRTCachingDirectory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/contrib/misc/src/java/org/apache/lucene/store/NRTCachingDirectory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/contrib/misc/src/java/org/apache/lucene/store/NRTCachingDirectory.java Mon Aug 22 12:07:31 2011
@@ -257,22 +257,6 @@ public class NRTCachingDirectory extends
     }
   }
 
-  // final due to LUCENE-3382: currently CFS backdoors the directory to create CFE
-  // by using the basic implementation and not delegating, we ensure that all 
-  // openInput/createOutput requests come thru NRTCachingDirectory.
-  @Override
-  public final CompoundFileDirectory openCompoundInput(String name, int bufferSize) throws IOException {
-    return super.openCompoundInput(name, bufferSize);
-  }
-  
-  // final due to LUCENE-3382: currently CFS backdoors the directory to create CFE
-  // by using the basic implementation and not delegating, we ensure that all 
-  // openInput/createOutput requests come thru NRTCachingDirectory.
-  @Override
-  public final CompoundFileDirectory createCompoundOutput(String name) throws IOException {
-    return super.createCompoundOutput(name);
-  }
-
   @Override
   public synchronized IndexInput openInput(String name, int bufferSize) throws IOException {
     if (cache.fileExists(name)) {

Modified: lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestIndexSplitter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestIndexSplitter.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestIndexSplitter.java (original)
+++ lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/index/TestIndexSplitter.java Mon Aug 22 12:07:31 2011
@@ -79,7 +79,7 @@ public class TestIndexSplitter extends L
     _TestUtil.rmDir(destDir2);
     destDir2.mkdirs();
     IndexSplitter.main(new String[] {dir.getAbsolutePath(), destDir2.getAbsolutePath(), splitSegName});
-    assertEquals(4, destDir2.listFiles().length);
+    assertEquals(3, destDir2.listFiles().length);
     Directory fsDirDest2 = newFSDirectory(destDir2);
     r = IndexReader.open(fsDirDest2, true);
     assertEquals(50, r.maxDoc());

Modified: lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/store/TestNRTCachingDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/store/TestNRTCachingDirectory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/store/TestNRTCachingDirectory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/contrib/misc/src/test/org/apache/lucene/store/TestNRTCachingDirectory.java Mon Aug 22 12:07:31 2011
@@ -146,32 +146,6 @@ public class TestNRTCachingDirectory ext
     }
   }
   
-  // LUCENE-3382 test that delegate compound files correctly.
-  public void testCompoundFileAppendTwice() throws IOException {
-    Directory newDir = new NRTCachingDirectory(newDirectory(), 2.0, 25.0);
-    CompoundFileDirectory csw = newDir.createCompoundOutput("d.cfs");
-    createSequenceFile(newDir, "d1", (byte) 0, 15);
-    IndexOutput out = csw.createOutput("d.xyz");
-    out.writeInt(0);
-    try {
-      newDir.copy(csw, "d1", "d1");
-      fail("file does already exist");
-    } catch (IOException e) {
-      //
-    }
-    out.close();
-    assertEquals(1, csw.listAll().length);
-    assertEquals("d.xyz", csw.listAll()[0]);
-   
-    csw.close();
-
-    CompoundFileDirectory cfr = newDir.openCompoundInput("d.cfs", 1024);
-    assertEquals(1, cfr.listAll().length);
-    assertEquals("d.xyz", cfr.listAll()[0]);
-    cfr.close();
-    newDir.close();
-  }
-  
   /** Creates a file of the specified size with sequential data. The first
    *  byte is written as the start byte provided. All subsequent bytes are
    *  computed as start + offset where offset is the number of the byte.

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/DocumentsWriter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/DocumentsWriter.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/DocumentsWriter.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/DocumentsWriter.java Mon Aug 22 12:07:31 2011
@@ -36,7 +36,6 @@ import org.apache.lucene.store.Directory
 import org.apache.lucene.store.RAMFile;
 import org.apache.lucene.util.ArrayUtil;
 import org.apache.lucene.util.BitVector;
-import org.apache.lucene.util.IOUtils;
 import org.apache.lucene.util.RamUsageEstimator;
 import org.apache.lucene.util.ThreadInterruptedException;
 
@@ -598,17 +597,11 @@ final class DocumentsWriter {
           message("flush: create compound file \"" + cfsFileName + "\"");
         }
 
-        final Directory cfsDir = directory.createCompoundOutput(cfsFileName);
-        IOException prior = null;
-        try {
-          for (String fileName : newSegment.files()) {
-            directory.copy(cfsDir, fileName, fileName);
-          }
-        } catch (IOException ex) {
-          prior = ex;
-        } finally {
-          IOUtils.closeSafely(prior, cfsDir);
+        CompoundFileWriter cfsWriter = new CompoundFileWriter(directory, cfsFileName);
+        for(String fileName : newSegment.files()) {
+          cfsWriter.addFile(fileName);
         }
+        cfsWriter.close();
         deleter.deleteNewFiles(newSegment.files());
         newSegment.setUseCompoundFile(true);
       }

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexFileNames.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexFileNames.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexFileNames.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexFileNames.java Mon Aug 22 12:07:31 2011
@@ -74,9 +74,6 @@ public final class IndexFileNames {
 
   /** Extension of compound file */
   public static final String COMPOUND_FILE_EXTENSION = "cfs";
-  
-  /** Extension of compound file entries */
-  public static final String COMPOUND_FILE_ENTRIES_EXTENSION = "cfe";
 
   /** Extension of compound file for doc store files*/
   public static final String COMPOUND_FILE_STORE_EXTENSION = "cfx";
@@ -106,7 +103,6 @@ public final class IndexFileNames {
    */
   public static final String INDEX_EXTENSIONS[] = new String[] {
     COMPOUND_FILE_EXTENSION,
-    COMPOUND_FILE_ENTRIES_EXTENSION,
     FIELD_INFOS_EXTENSION,
     FIELDS_INDEX_EXTENSION,
     FIELDS_EXTENSION,
@@ -266,15 +262,7 @@ public final class IndexFileNames {
     }
     return filename;
   }
-  
-  public static String stripExtension(String filename) {
-    int idx = filename.indexOf('.');
-    if (idx != -1) {
-      filename = filename.substring(0, idx);
-    }
-    return filename;
-  }
-  
+
   /**
    * Returns true if the given filename ends with the separate norms file
    * pattern: {@code SEPARATE_NORMS_EXTENSION + "[0-9]+"}.

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexReader.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexReader.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexReader.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexReader.java Mon Aug 22 12:07:31 2011
@@ -1254,14 +1254,14 @@ public abstract class IndexReader implem
     }
 
     Directory dir = null;
-    CompoundFileDirectory cfr = null;
+    CompoundFileReader cfr = null;
 
     try {
       File file = new File(filename);
       String dirname = file.getAbsoluteFile().getParent();
       filename = file.getName();
       dir = FSDirectory.open(new File(dirname));
-      cfr = dir.openCompoundInput(filename, BufferedIndexInput.BUFFER_SIZE);
+      cfr = new CompoundFileReader(dir, filename);
 
       String [] files = cfr.listAll();
       ArrayUtil.mergeSort(files);   // sort the array of filename so that the output is more readable

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexWriter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexWriter.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexWriter.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/IndexWriter.java Mon Aug 22 12:07:31 2011
@@ -48,7 +48,6 @@ import org.apache.lucene.store.Directory
 import org.apache.lucene.store.Lock;
 import org.apache.lucene.store.LockObtainFailedException;
 import org.apache.lucene.util.Constants;
-import org.apache.lucene.util.IOUtils;
 import org.apache.lucene.util.StringHelper;
 import org.apache.lucene.util.ThreadInterruptedException;
 import org.apache.lucene.util.Version;
@@ -1209,7 +1208,7 @@ public class IndexWriter implements Clos
     Directory cfsDir = null;
     try {
       if (info.getUseCompoundFile()) {
-        cfsDir = directory.openCompoundInput(IndexFileNames.segmentFileName(info.name, IndexFileNames.COMPOUND_FILE_EXTENSION), 1024);
+        cfsDir = new CompoundFileReader(directory, IndexFileNames.segmentFileName(info.name, IndexFileNames.COMPOUND_FILE_EXTENSION));
       } else {
         cfsDir = directory;
       }
@@ -3236,24 +3235,21 @@ public class IndexWriter implements Clos
   private void copySegmentIntoCFS(SegmentInfo info, String segName) throws IOException {
     String segFileName = IndexFileNames.segmentFileName(segName, IndexFileNames.COMPOUND_FILE_EXTENSION);
     Collection<String> files = info.files();
-    final Directory cfsDir = directory.createCompoundOutput(segFileName);
-    IOException prior = null;
-    try {
-      for (String file : files) {
-        String newFileName = segName + IndexFileNames.stripSegmentName(file);
-        if (!IndexFileNames.matchesExtension(file, IndexFileNames.DELETES_EXTENSION)
-            && !IndexFileNames.isSeparateNormsFile(file)) {
-          info.dir.copy(cfsDir, file, file);
-        } else {
-          assert !directory.fileExists(newFileName): "file \"" + newFileName + "\" already exists";
-          info.dir.copy(directory, file, newFileName);
-        }
+    CompoundFileWriter cfsWriter = new CompoundFileWriter(directory, segFileName);
+    for (String file : files) {
+      String newFileName = segName + IndexFileNames.stripSegmentName(file);
+      if (!IndexFileNames.matchesExtension(file, IndexFileNames.DELETES_EXTENSION)
+          && !IndexFileNames.isSeparateNormsFile(file)) {
+        cfsWriter.addFile(file, info.dir);
+      } else {
+        assert !directory.fileExists(newFileName): "file \"" + newFileName + "\" already exists";
+        info.dir.copy(directory, file, newFileName);
       }
-    } catch(IOException ex) {
-      prior = ex;
-    } finally {
-      IOUtils.closeSafely(prior, cfsDir);
     }
+    
+    // Create the .cfs
+    cfsWriter.close();
+    
     info.dir = directory;
     info.name = segName;
     info.setUseCompoundFile(true);

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentCoreReaders.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentCoreReaders.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentCoreReaders.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentCoreReaders.java Mon Aug 22 12:07:31 2011
@@ -20,7 +20,6 @@ package org.apache.lucene.index;
 import java.io.IOException;
 import java.util.concurrent.atomic.AtomicInteger;
 
-import org.apache.lucene.store.CompoundFileDirectory;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.IndexInput;
 
@@ -52,8 +51,8 @@ final class SegmentCoreReaders {
   TermInfosReader tis;
   FieldsReader fieldsReaderOrig;
   TermVectorsReader termVectorsReaderOrig;
-  CompoundFileDirectory cfsReader;
-  CompoundFileDirectory storeCFSReader;
+  CompoundFileReader cfsReader;
+  CompoundFileReader storeCFSReader;
 
   SegmentCoreReaders(SegmentReader owner, Directory dir, SegmentInfo si, int readBufferSize, int termsIndexDivisor) throws IOException {
     segment = si.name;
@@ -65,7 +64,7 @@ final class SegmentCoreReaders {
     try {
       Directory dir0 = dir;
       if (si.getUseCompoundFile()) {
-        cfsReader = dir.openCompoundInput(IndexFileNames.segmentFileName(segment, IndexFileNames.COMPOUND_FILE_EXTENSION), readBufferSize);
+        cfsReader = new CompoundFileReader(dir, IndexFileNames.segmentFileName(segment, IndexFileNames.COMPOUND_FILE_EXTENSION), readBufferSize);
         dir0 = cfsReader;
       }
       cfsDir = dir0;
@@ -146,7 +145,7 @@ final class SegmentCoreReaders {
         // terms reader with index, the segment has switched
         // to CFS
         if (cfsReader == null) {
-          cfsReader = dir.openCompoundInput(IndexFileNames.segmentFileName(segment, IndexFileNames.COMPOUND_FILE_EXTENSION), readBufferSize);
+          cfsReader = new CompoundFileReader(dir, IndexFileNames.segmentFileName(segment, IndexFileNames.COMPOUND_FILE_EXTENSION), readBufferSize);
         }
         dir0 = cfsReader;
       } else {
@@ -212,7 +211,7 @@ final class SegmentCoreReaders {
       if (si.getDocStoreOffset() != -1) {
         if (si.getDocStoreIsCompoundFile()) {
           assert storeCFSReader == null;
-          storeCFSReader = dir.openCompoundInput(
+          storeCFSReader = new CompoundFileReader(dir,
               IndexFileNames.segmentFileName(si.getDocStoreSegment(), IndexFileNames.COMPOUND_FILE_STORE_EXTENSION),
                                                   readBufferSize);
           storeDir = storeCFSReader;
@@ -226,7 +225,7 @@ final class SegmentCoreReaders {
         // was not used, but then we are asked to open doc
         // stores after the segment has switched to CFS
         if (cfsReader == null) {
-          cfsReader = dir.openCompoundInput(IndexFileNames.segmentFileName(segment, IndexFileNames.COMPOUND_FILE_EXTENSION), readBufferSize);
+          cfsReader = new CompoundFileReader(dir, IndexFileNames.segmentFileName(segment, IndexFileNames.COMPOUND_FILE_EXTENSION), readBufferSize);
         }
         storeDir = cfsReader;
         assert storeDir != null;

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfo.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfo.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfo.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfo.java Mon Aug 22 12:07:31 2011
@@ -22,7 +22,6 @@ import org.apache.lucene.store.IndexOutp
 import org.apache.lucene.store.IndexInput;
 import org.apache.lucene.util.BitVector;
 import org.apache.lucene.util.Constants;
-import org.apache.lucene.util.StringHelper;
 
 import java.io.IOException;
 import java.util.HashSet;
@@ -238,7 +237,7 @@ public final class SegmentInfo implement
         }
         final Directory dirToTest;
         if (isCompoundFile) {
-          dirToTest = dir.openCompoundInput(IndexFileNames.segmentFileName(storesSegment, ext), 1024);
+          dirToTest = new CompoundFileReader(dir, IndexFileNames.segmentFileName(storesSegment, ext));
         } else {
           dirToTest = dir;
         }
@@ -658,10 +657,6 @@ public final class SegmentInfo implement
 
     if (useCompoundFile) {
       filesSet.add(IndexFileNames.segmentFileName(name, IndexFileNames.COMPOUND_FILE_EXTENSION));
-      if (version != null && StringHelper.getVersionComparator().compare("3.4", version) <= 0) {
-        filesSet.add(IndexFileNames.segmentFileName(name,
-            IndexFileNames.COMPOUND_FILE_ENTRIES_EXTENSION));
-      }
     } else {
       for (String ext : IndexFileNames.NON_STORE_INDEX_EXTENSIONS)
         addIfExists(filesSet, IndexFileNames.segmentFileName(name, ext));

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfos.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfos.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfos.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentInfos.java Mon Aug 22 12:07:31 2011
@@ -289,12 +289,12 @@ public final class SegmentInfos implemen
           Directory dir = directory;
           if (si.getDocStoreOffset() != -1) {
             if (si.getDocStoreIsCompoundFile()) {
-              dir = dir.openCompoundInput(IndexFileNames.segmentFileName(
+              dir = new CompoundFileReader(dir, IndexFileNames.segmentFileName(
                   si.getDocStoreSegment(),
                   IndexFileNames.COMPOUND_FILE_STORE_EXTENSION), 1024);
             }
           } else if (si.getUseCompoundFile()) {
-            dir = dir.openCompoundInput(IndexFileNames.segmentFileName(
+            dir = new CompoundFileReader(dir, IndexFileNames.segmentFileName(
                 si.name, IndexFileNames.COMPOUND_FILE_EXTENSION), 1024);
           }
 

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentMerger.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentMerger.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentMerger.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/index/SegmentMerger.java Mon Aug 22 12:07:31 2011
@@ -27,7 +27,6 @@ import org.apache.lucene.index.FieldInfo
 import org.apache.lucene.index.IndexReader.FieldOption;
 import org.apache.lucene.index.MergePolicy.MergeAbortedException;
 import org.apache.lucene.index.PayloadProcessorProvider.PayloadProcessor;
-import org.apache.lucene.store.CompoundFileDirectory;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.IndexInput;
 import org.apache.lucene.store.IndexOutput;
@@ -126,19 +125,18 @@ final class SegmentMerger {
           throws IOException {
     // Now merge all added files
     Collection<String> files = info.files();
-    CompoundFileDirectory cfsDir = directory.createCompoundOutput(fileName);
-    try {
-      for (String file : files) {
-        assert !IndexFileNames.matchesExtension(file, IndexFileNames.DELETES_EXTENSION) 
-                  : ".del file is not allowed in .cfs: " + file;
-        assert !IndexFileNames.isSeparateNormsFile(file) 
-                  : "separate norms file (.s[0-9]+) is not allowed in .cfs: " + file;
-        directory.copy(cfsDir, file, file);
-        checkAbort.work(directory.fileLength(file));
-      }
-    } finally {
-      cfsDir.close();
+    CompoundFileWriter cfsWriter = new CompoundFileWriter(directory, fileName, checkAbort);
+    for (String file : files) {
+      assert !IndexFileNames.matchesExtension(file, IndexFileNames.DELETES_EXTENSION) 
+                : ".del file is not allowed in .cfs: " + file;
+      assert !IndexFileNames.isSeparateNormsFile(file)
+                : "separate norms file (.s[0-9]+) is not allowed in .cfs: " + file;
+      cfsWriter.addFile(file);
     }
+    
+    // Perform the merge
+    cfsWriter.close();
+   
     return files;
   }
 

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/DataOutput.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/DataOutput.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/DataOutput.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/DataOutput.java Mon Aug 22 12:07:31 2011
@@ -60,14 +60,6 @@ public abstract class DataOutput {
     writeByte((byte)(i >>  8));
     writeByte((byte) i);
   }
-  
-  /** Writes a short as two bytes.
-   * @see DataInput#readShort()
-   */
-  public void writeShort(short i) throws IOException {
-    writeByte((byte)(i >>  8));
-    writeByte((byte) i);
-  }
 
   /** Writes an int in a variable-length format.  Writes between one and
    * five bytes.  Smaller values take fewer bytes.  Negative numbers are not

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/Directory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/Directory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/Directory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/Directory.java Mon Aug 22 12:07:31 2011
@@ -137,35 +137,11 @@ public abstract class Directory implemen
    * implementation may ignore the buffer size.  Currently
    * the only Directory implementations that respect this
    * parameter are {@link FSDirectory} and {@link
-   * CompoundFileDirectory}.
+   * org.apache.lucene.index.CompoundFileReader}.
   */
   public IndexInput openInput(String name, int bufferSize) throws IOException {
     return openInput(name);
   }
-  
-  /** 
-   * Returns a {@link CompoundFileDirectory} capable of
-   * reading the Lucene compound file format.  
-   * <p>
-   * The default implementation returns 
-   * {@link DefaultCompoundFileDirectory}.
-   * @lucene.experimental
-   */
-  public CompoundFileDirectory openCompoundInput(String name, int bufferSize) throws IOException {
-    return new DefaultCompoundFileDirectory(this, name, bufferSize, false);
-  }
-  
-  /** 
-   * Returns a {@link CompoundFileDirectory} capable of
-   * writing the Lucene compound file format.  
-   * <p>
-   * The default implementation returns 
-   * {@link DefaultCompoundFileDirectory}.
-   * @lucene.experimental
-   */
-  public CompoundFileDirectory createCompoundOutput(String name) throws IOException {
-    return new DefaultCompoundFileDirectory(this, name, 1024, true);
-  }
 
   /** Construct a {@link Lock}.
    * @param name the name of the lock file

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/FileSwitchDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/FileSwitchDirectory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/FileSwitchDirectory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/FileSwitchDirectory.java Mon Aug 22 12:07:31 2011
@@ -190,20 +190,4 @@ public class FileSwitchDirectory extends
   public IndexInput openInput(String name) throws IOException {
     return getDirectory(name).openInput(name);
   }
-
-  // final due to LUCENE-3380: currently CFS backdoors the directory to create CFE
-  // by using the basic implementation and not delegating, we ensure that all 
-  // openInput/createOutput requests come thru NRTCachingDirectory.
-  @Override
-  public final CompoundFileDirectory openCompoundInput(String name, int bufferSize) throws IOException {
-    return super.openCompoundInput(name, bufferSize);
-  }
-  
-  // final due to LUCENE-3380: currently CFS backdoors the directory to create CFE
-  // by using the basic implementation and not delegating, we ensure that all 
-  // openInput/createOutput requests come thru NRTCachingDirectory.
-  @Override
-  public final CompoundFileDirectory createCompoundOutput(String name) throws IOException {
-    return super.createCompoundOutput(name);
-  }
 }

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/IndexOutput.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/IndexOutput.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/IndexOutput.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/IndexOutput.java Mon Aug 22 12:07:31 2011
@@ -58,5 +58,4 @@ public abstract class IndexOutput extend
    * @param length file length
    */
   public void setLength(long length) throws IOException {}
-
 }

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/MMapDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/MMapDirectory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/MMapDirectory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/MMapDirectory.java Mon Aug 22 12:07:31 2011
@@ -32,7 +32,6 @@ import java.security.PrivilegedActionExc
 import java.lang.reflect.Method;
 
 import org.apache.lucene.util.Constants;
-import org.apache.lucene.util.IOUtils;
 
 /** File-based {@link Directory} implementation that uses
  *  mmap for reading, and {@link
@@ -214,49 +213,11 @@ public class MMapDirectory extends FSDir
     File f = new File(getDirectory(), name);
     RandomAccessFile raf = new RandomAccessFile(f, "r");
     try {
-      return new MMapIndexInput(raf, 0, raf.length(), chunkSizePower);
+      return new MMapIndexInput(raf, chunkSizePower);
     } finally {
       raf.close();
     }
   }
-  
-  @Override
-  public CompoundFileDirectory openCompoundInput(String name, int bufferSize) throws IOException {
-    return new MMapCompoundFileDirectory(name, bufferSize);
-  }
-  
-  private final class MMapCompoundFileDirectory extends CompoundFileDirectory {
-    private RandomAccessFile raf = null;
-
-    public MMapCompoundFileDirectory(String fileName, int readBufferSize) throws IOException {
-      super(MMapDirectory.this, fileName, readBufferSize);
-      IndexInput stream = null;
-      try {
-        File f = new File(MMapDirectory.this.getDirectory(), fileName);
-        raf = new RandomAccessFile(f, "r");
-        stream = new MMapIndexInput(raf, 0, raf.length(), chunkSizePower);
-        initForRead(CompoundFileDirectory.readEntries(stream, MMapDirectory.this, fileName));
-        stream.close();
-      } catch (IOException e) {
-        // throw our original exception
-        IOUtils.closeSafely(e, raf, stream);
-      }
-    }
-
-    @Override
-    public IndexInput openInputSlice(String id, long offset, long length, int readBufferSize) throws IOException {
-      return new MMapIndexInput(raf, offset, length, chunkSizePower);
-    }
-
-    @Override
-    public synchronized void close() throws IOException {
-      try {
-        raf.close();
-      } finally {
-        super.close();
-      }
-    }
-  }
 
   // Because Java's ByteBuffer uses an int to address the
   // values, it's necessary to access a file >
@@ -274,8 +235,8 @@ public class MMapDirectory extends FSDir
   
     private boolean isClone = false;
     
-    MMapIndexInput(RandomAccessFile raf, long offset, long length, int chunkSizePower) throws IOException {
-      this.length = length;
+    MMapIndexInput(RandomAccessFile raf, int chunkSizePower) throws IOException {
+      this.length = raf.length();
       this.chunkSizePower = chunkSizePower;
       this.chunkSize = 1L << chunkSizePower;
       this.chunkSizeMask = chunkSize - 1L;
@@ -300,7 +261,7 @@ public class MMapDirectory extends FSDir
           ? chunkSize
           : (length - bufferStart)
         );
-        this.buffers[bufNr] = rafc.map(MapMode.READ_ONLY, offset + bufferStart, bufSize);
+        this.buffers[bufNr] = rafc.map(MapMode.READ_ONLY, bufferStart, bufSize);
         bufferStart += bufSize;
       }
       seek(0L);

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/NIOFSDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/NIOFSDirectory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/NIOFSDirectory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/NIOFSDirectory.java Mon Aug 22 12:07:31 2011
@@ -24,9 +24,6 @@ import java.nio.channels.ClosedChannelEx
 import java.nio.channels.FileChannel;
 import java.util.concurrent.Future; // javadoc
 
-import org.apache.lucene.store.SimpleFSDirectory.SimpleFSIndexInput;
-import org.apache.lucene.util.IOUtils;
-
 /**
  * An {@link FSDirectory} implementation that uses java.nio's FileChannel's
  * positional read, which allows multiple threads to read from the same file
@@ -80,47 +77,6 @@ public class NIOFSDirectory extends FSDi
     ensureOpen();
     return new NIOFSIndexInput(new File(getDirectory(), name), bufferSize, getReadChunkSize());
   }
-  
-  @Override
-  public CompoundFileDirectory openCompoundInput(String name, int bufferSize) throws IOException {
-    return new NIOFSCompoundFileDirectory(name, bufferSize);
-  }
-
-  private final class NIOFSCompoundFileDirectory extends CompoundFileDirectory {
-    private SimpleFSIndexInput.Descriptor fd;
-    private FileChannel fc;
-
-    public NIOFSCompoundFileDirectory(String fileName, int readBufferSize) throws IOException {
-      super(NIOFSDirectory.this, fileName, readBufferSize);
-      IndexInput stream = null;
-      try {
-        File f = new File(NIOFSDirectory.this.getDirectory(), fileName);
-        fd = new SimpleFSIndexInput.Descriptor(f, "r");
-        fc = fd.getChannel();
-        stream = new NIOFSIndexInput(fd, fc, 0, fd.length, readBufferSize,
-            getReadChunkSize());
-        initForRead(CompoundFileDirectory.readEntries(stream, NIOFSDirectory.this, fileName));
-        stream.close();
-      } catch (IOException e) {
-        // throw our original exception
-        IOUtils.closeSafely(e, fc, fd, stream);
-      }
-    }
-    
-    @Override
-    public IndexInput openInputSlice(String id, long offset, long length, int readBufferSize) throws IOException {
-      return new NIOFSIndexInput(fd, fc, offset, length, readBufferSize, getReadChunkSize());
-    }
-
-    @Override
-    public synchronized void close() throws IOException {
-      try {
-        IOUtils.closeSafely(false, fc, fd);
-      } finally {
-        super.close();
-      }
-    }
-  }
 
   protected static class NIOFSIndexInput extends SimpleFSDirectory.SimpleFSIndexInput {
 
@@ -135,12 +91,6 @@ public class NIOFSDirectory extends FSDi
       super(path, bufferSize, chunkSize);
       channel = file.getChannel();
     }
-    
-    public NIOFSIndexInput(Descriptor file, FileChannel fc, long off, long length, int bufferSize, int chunkSize) throws IOException {
-      super(file, off, length, bufferSize, chunkSize);
-      channel = fc;
-      isClone = true;
-    }
 
     @Override
     protected void newBuffer(byte[] newBuffer) {
@@ -195,11 +145,7 @@ public class NIOFSDirectory extends FSDi
       int readLength = bb.limit() - readOffset;
       assert readLength == len;
 
-      long pos = getFilePointer() + off;
-      
-      if (pos + len > end) {
-        throw new IOException("read past EOF");
-      }
+      long pos = getFilePointer();
 
       try {
         while (readLength > 0) {
@@ -213,6 +159,9 @@ public class NIOFSDirectory extends FSDi
           }
           bb.limit(limit);
           int i = channel.read(bb, pos);
+          if (i == -1) {
+            throw new IOException("read past EOF");
+          }
           pos += i;
           readOffset += i;
           readLength -= i;

Modified: lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/SimpleFSDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/SimpleFSDirectory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/SimpleFSDirectory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/store/SimpleFSDirectory.java Mon Aug 22 12:07:31 2011
@@ -21,8 +21,6 @@ import java.io.File;
 import java.io.IOException;
 import java.io.RandomAccessFile;
 
-import org.apache.lucene.util.IOUtils;
-
 /** A straightforward implementation of {@link FSDirectory}
  *  using java.io.RandomAccessFile.  However, this class has
  *  poor concurrent performance (multiple threads will
@@ -57,45 +55,6 @@ public class SimpleFSDirectory extends F
     ensureOpen();
     return new SimpleFSIndexInput(new File(directory, name), bufferSize, getReadChunkSize());
   }
-  
-  @Override
-  public CompoundFileDirectory openCompoundInput(String name, int bufferSize) throws IOException {
-    return new SimpleFSCompoundFileDirectory(name, bufferSize);
-  }
-
-  private final class SimpleFSCompoundFileDirectory extends CompoundFileDirectory {
-    private SimpleFSIndexInput.Descriptor fd;
-
-    public SimpleFSCompoundFileDirectory(String fileName, int readBufferSize) throws IOException {
-      super(SimpleFSDirectory.this, fileName, readBufferSize);
-      IndexInput stream = null;
-      try {
-        final File f = new File(SimpleFSDirectory.this.getDirectory(), fileName);
-        fd = new SimpleFSIndexInput.Descriptor(f, "r");
-        stream = new SimpleFSIndexInput(fd, 0, fd.length, readBufferSize,
-            getReadChunkSize());
-        initForRead(CompoundFileDirectory.readEntries(stream, SimpleFSDirectory.this, fileName));
-        stream.close();
-      } catch (IOException e) {
-        // throw our original exception
-        IOUtils.closeSafely(e, fd, stream);
-      }
-    }
-
-    @Override
-    public IndexInput openInputSlice(String id, long offset, long length, int readBufferSize) throws IOException {
-      return new SimpleFSIndexInput(fd, offset, length, readBufferSize, getReadChunkSize());
-    }
-
-    @Override
-    public synchronized void close() throws IOException {
-      try {
-        fd.close();
-      } finally {
-        super.close();
-      }
-    }
-  }
 
   protected static class SimpleFSIndexInput extends BufferedIndexInput {
   
@@ -125,24 +84,11 @@ public class SimpleFSDirectory extends F
     boolean isClone;
     //  LUCENE-1566 - maximum read length on a 32bit JVM to prevent incorrect OOM 
     protected final int chunkSize;
-    protected final long off;
-    protected final long end;
     
     public SimpleFSIndexInput(File path, int bufferSize, int chunkSize) throws IOException {
       super(bufferSize);
-      this.file = new Descriptor(path, "r"); 
+      file = new Descriptor(path, "r");
       this.chunkSize = chunkSize;
-      this.off = 0L;
-      this.end = file.length;
-    }
-    
-    public SimpleFSIndexInput(Descriptor file, long off, long length, int bufferSize, int chunkSize) throws IOException {
-      super(bufferSize);
-      this.file = file;
-      this.chunkSize = chunkSize;
-      this.off = off;
-      this.end = off + length;
-      this.isClone = true; // well, we are sorta?
     }
   
     /** IndexInput methods */
@@ -150,17 +96,13 @@ public class SimpleFSDirectory extends F
     protected void readInternal(byte[] b, int offset, int len)
          throws IOException {
       synchronized (file) {
-        long position = off + getFilePointer();
+        long position = getFilePointer();
         if (position != file.position) {
           file.seek(position);
           file.position = position;
         }
         int total = 0;
 
-        if (position + len > end) {
-          throw new IOException("read past EOF");
-        }
-
         try {
           do {
             final int readLength;
@@ -171,6 +113,9 @@ public class SimpleFSDirectory extends F
               readLength = chunkSize;
             }
             final int i = file.read(b, offset + total, readLength);
+            if (i == -1) {
+              throw new IOException("read past EOF");
+            }
             file.position += i;
             total += i;
           } while (total < len);
@@ -199,7 +144,7 @@ public class SimpleFSDirectory extends F
   
     @Override
     public long length() {
-      return end - off;
+      return file.length;
     }
   
     @Override

Modified: lucene/dev/branches/branch_3x/lucene/src/test-framework/org/apache/lucene/store/MockDirectoryWrapper.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/test-framework/org/apache/lucene/store/MockDirectoryWrapper.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/test-framework/org/apache/lucene/store/MockDirectoryWrapper.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/test-framework/org/apache/lucene/store/MockDirectoryWrapper.java Mon Aug 22 12:07:31 2011
@@ -426,18 +426,6 @@ public class MockDirectoryWrapper extend
     addFileHandle(ii, name, true);
     return ii;
   }
-  
-  @Override
-  public synchronized CompoundFileDirectory openCompoundInput(String name, int bufferSize) throws IOException {
-    maybeYield();
-    return new MockCompoundFileDirectoryWrapper(name, this, delegate.openCompoundInput(name, bufferSize), false);
-  }
-   
-  @Override
-  public CompoundFileDirectory createCompoundOutput(String name) throws IOException {
-    maybeYield();
-    return new MockCompoundFileDirectoryWrapper(name, this, delegate.createCompoundOutput(name), true);
-  }
 
   /** Provided for testing purposes.  Use sizeInBytes() instead. */
   public synchronized final long getRecomputedSizeInBytes() throws IOException {

Modified: lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestAddIndexes.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestAddIndexes.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestAddIndexes.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestAddIndexes.java Mon Aug 22 12:07:31 2011
@@ -949,7 +949,7 @@ public class TestAddIndexes extends Luce
     w3.addIndexes(readers);
     w3.close();
     
-    assertEquals("Only one compound segment should exist", 4, dir.listAll().length);
+    assertEquals("Only one compound segment should exist", 3, dir.listAll().length);
   }
  
   // LUCENE-2996: tests that addIndexes(IndexReader) applies existing deletes correctly.

Modified: lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java Mon Aug 22 12:07:31 2011
@@ -24,6 +24,7 @@ import java.io.DataInputStream;
 import java.util.Arrays;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.ArrayList;
 import java.util.Random;
 
 import org.apache.lucene.analysis.WhitespaceAnalyzer;
@@ -563,7 +564,7 @@ public class TestBackwardsCompatibility 
       // figure out which field number corresponds to
       // "content", and then set our expected file names below
       // accordingly:
-      Directory cfsReader = dir.openCompoundInput("_0.cfs", 1024);
+      CompoundFileReader cfsReader = new CompoundFileReader(dir, "_0.cfs");
       FieldInfos fieldInfos = new FieldInfos(cfsReader, "_0.fnm");
       int contentFieldIndex = -1;
       for(int i=0;i<fieldInfos.size();i++) {
@@ -578,7 +579,6 @@ public class TestBackwardsCompatibility 
 
       // Now verify file names:
       String[] expected = new String[] {"_0.cfs",
-                               "_0.cfe",
                                "_0_1.del",
                                "_0_1.s" + contentFieldIndex,
                                "segments_2",

Modified: lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestCompoundFile.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestCompoundFile.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestCompoundFile.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestCompoundFile.java Mon Aug 22 12:07:31 2011
@@ -21,8 +21,6 @@ import java.io.IOException;
 import java.io.File;
 
 import org.apache.lucene.util.LuceneTestCase;
-
-import org.apache.lucene.store.CompoundFileDirectory;
 import org.apache.lucene.store.IndexOutput;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.IndexInput;
@@ -183,11 +181,11 @@ public class TestCompoundFile extends Lu
         for (int i=0; i<data.length; i++) {
             String name = "t" + data[i];
             createSequenceFile(dir, name, (byte) 0, data[i]);
-            CompoundFileDirectory csw = dir.createCompoundOutput(name + ".cfs");
-            dir.copy(csw, name, name);
+            CompoundFileWriter csw = new CompoundFileWriter(dir, name + ".cfs");
+            csw.addFile(name);
             csw.close();
 
-            CompoundFileDirectory csr = dir.openCompoundInput(name + ".cfs", 1024);
+            CompoundFileReader csr = new CompoundFileReader(dir, name + ".cfs");
             IndexInput expected = dir.openInput(name);
             IndexInput actual = csr.openInput(name);
             assertSameStreams(name, expected, actual);
@@ -206,12 +204,12 @@ public class TestCompoundFile extends Lu
         createSequenceFile(dir, "d1", (byte) 0, 15);
         createSequenceFile(dir, "d2", (byte) 0, 114);
 
-        CompoundFileDirectory csw = dir.createCompoundOutput("d.cfs");
-        dir.copy(csw, "d1", "d1");
-        dir.copy(csw, "d2", "d2");
+        CompoundFileWriter csw = new CompoundFileWriter(dir, "d.csf");
+        csw.addFile("d1");
+        csw.addFile("d2");
         csw.close();
 
-        CompoundFileDirectory csr = dir.openCompoundInput("d.cfs", 1024);
+        CompoundFileReader csr = new CompoundFileReader(dir, "d.csf");
         IndexInput expected = dir.openInput("d1");
         IndexInput actual = csr.openInput("d1");
         assertSameStreams("d1", expected, actual);
@@ -256,18 +254,17 @@ public class TestCompoundFile extends Lu
         createRandomFile(dir, segment + ".notIn2", 51);
 
         // Now test
-        CompoundFileDirectory csw = dir.createCompoundOutput("test.cfs");
+        CompoundFileWriter csw = new CompoundFileWriter(dir, "test.cfs");
         final String data[] = new String[] {
             ".zero", ".one", ".ten", ".hundred", ".big1", ".big2", ".big3",
             ".big4", ".big5", ".big6", ".big7"
         };
         for (int i=0; i<data.length; i++) {
-            String fileName = segment + data[i];
-            dir.copy(csw, fileName, fileName);
+            csw.addFile(segment + data[i]);
         }
         csw.close();
 
-        CompoundFileDirectory csr = dir.openCompoundInput("test.cfs", 1024);
+        CompoundFileReader csr = new CompoundFileReader(dir, "test.cfs");
         for (int i=0; i<data.length; i++) {
             IndexInput check = dir.openInput(segment + data[i]);
             IndexInput test = csr.openInput(segment + data[i]);
@@ -286,11 +283,10 @@ public class TestCompoundFile extends Lu
      *  the size of each file is 1000 bytes.
      */
     private void setUp_2() throws IOException {
-        CompoundFileDirectory cw = dir.createCompoundOutput("f.comp");
+        CompoundFileWriter cw = new CompoundFileWriter(dir, "f.comp");
         for (int i=0; i<20; i++) {
             createSequenceFile(dir, "f" + i, (byte) 0, 2000);
-            String fileName = "f" + i;
-            dir.copy(cw, fileName, fileName);
+            cw.addFile("f" + i);
         }
         cw.close();
     }
@@ -335,9 +331,26 @@ public class TestCompoundFile extends Lu
         }
     }
 
+
+    static boolean isCSIndexInput(IndexInput is) {
+        return is instanceof CompoundFileReader.CSIndexInput;
+    }
+
+    static boolean isCSIndexInputOpen(IndexInput is) throws IOException {
+        if (isCSIndexInput(is)) {
+            CompoundFileReader.CSIndexInput cis =
+            (CompoundFileReader.CSIndexInput) is;
+
+            return _TestHelper.isSimpleFSIndexInputOpen(cis.base);
+        } else {
+            return false;
+        }
+    }
+
+
     public void testClonedStreamsClosing() throws IOException {
         setUp_2();
-        CompoundFileDirectory cr = dir.openCompoundInput("f.comp", 1024);
+        CompoundFileReader cr = new CompoundFileReader(dir, "f.comp");
 
         // basic clone
         IndexInput expected = dir.openInput("f11");
@@ -347,8 +360,10 @@ public class TestCompoundFile extends Lu
         assertTrue(_TestHelper.isSimpleFSIndexInputOpen(expected));
 
         IndexInput one = cr.openInput("f11");
+        assertTrue(isCSIndexInputOpen(one));
 
         IndexInput two = (IndexInput) one.clone();
+        assertTrue(isCSIndexInputOpen(two));
 
         assertSameStreams("basic clone one", expected, one);
         expected.seek(0);
@@ -356,6 +371,7 @@ public class TestCompoundFile extends Lu
 
         // Now close the first stream
         one.close();
+        assertTrue("Only close when cr is closed", isCSIndexInputOpen(one));
 
         // The following should really fail since we couldn't expect to
         // access a file once close has been called on it (regardless of
@@ -367,6 +383,8 @@ public class TestCompoundFile extends Lu
 
         // Now close the compound reader
         cr.close();
+        assertFalse("Now closed one", isCSIndexInputOpen(one));
+        assertFalse("Now closed two", isCSIndexInputOpen(two));
 
         // The following may also fail since the compound stream is closed
         expected.seek(0);
@@ -389,7 +407,7 @@ public class TestCompoundFile extends Lu
      */
     public void testRandomAccess() throws IOException {
         setUp_2();
-        CompoundFileDirectory cr = dir.openCompoundInput("f.comp", 1024);
+        CompoundFileReader cr = new CompoundFileReader(dir, "f.comp");
 
         // Open two files
         IndexInput e1 = dir.openInput("f11");
@@ -468,7 +486,7 @@ public class TestCompoundFile extends Lu
      */
     public void testRandomAccessClones() throws IOException {
         setUp_2();
-        CompoundFileDirectory cr = dir.openCompoundInput("f.comp", 1024);
+        CompoundFileReader cr = new CompoundFileReader(dir, "f.comp");
 
         // Open two files
         IndexInput e1 = cr.openInput("f11");
@@ -545,7 +563,7 @@ public class TestCompoundFile extends Lu
 
     public void testFileNotFound() throws IOException {
         setUp_2();
-        CompoundFileDirectory cr = dir.openCompoundInput("f.comp", 1024);
+        CompoundFileReader cr = new CompoundFileReader(dir, "f.comp");
 
         // Open two files
         try {
@@ -563,7 +581,7 @@ public class TestCompoundFile extends Lu
 
     public void testReadPastEOF() throws IOException {
         setUp_2();
-        CompoundFileDirectory cr = dir.openCompoundInput("f.comp", 1024);
+        CompoundFileReader cr = new CompoundFileReader(dir, "f.comp");
         IndexInput is = cr.openInput("f2");
         is.seek(is.length() - 10);
         byte b[] = new byte[100];
@@ -616,11 +634,11 @@ public class TestCompoundFile extends Lu
        createSequenceFile(dir, "d1", (byte) 0, 15);
 
        Directory newDir = newDirectory();
-       CompoundFileDirectory csw = newDir.createCompoundOutput("d.cfs");
-       dir.copy(csw, "d1", "d1");
+       CompoundFileWriter csw = new CompoundFileWriter(newDir, "d.csf");
+       csw.addFile("d1", dir);
        csw.close();
 
-       CompoundFileDirectory csr = newDir.openCompoundInput("d.cfs", 1024);
+       CompoundFileReader csr = new CompoundFileReader(newDir, "d.csf");
        IndexInput expected = dir.openInput("d1");
        IndexInput actual = csr.openInput("d1");
        assertSameStreams("d1", expected, actual);
@@ -631,83 +649,4 @@ public class TestCompoundFile extends Lu
        
        newDir.close();
    }
-   
-   
-  public void testAppend() throws IOException {
-    Directory newDir = newDirectory();
-    CompoundFileDirectory csw = newDir.createCompoundOutput("d.cfs");
-    int size = 5 + random.nextInt(128);
-    for (int j = 0; j < 2; j++) {
-      IndexOutput os = csw.createOutput("seg" + j + "_foo.txt");
-      for (int i = 0; i < size; i++) {
-        os.writeInt(i);
-      }
-      os.close();
-      String[] listAll = newDir.listAll();
-      assertEquals(1, listAll.length);
-      assertEquals("d.cfs", listAll[0]);
-    }
-    createSequenceFile(dir, "d1", (byte) 0, 15);
-    dir.copy(csw, "d1", "d1");
-    String[] listAll = newDir.listAll();
-    assertEquals(1, listAll.length);
-    assertEquals("d.cfs", listAll[0]);
-    csw.close();
-    CompoundFileDirectory csr = newDir.openCompoundInput("d.cfs", 1024);
-    for (int j = 0; j < 2; j++) {
-      IndexInput openInput = csr.openInput("seg" + j + "_foo.txt");
-      assertEquals(size * 4, openInput.length());
-      for (int i = 0; i < size; i++) {
-        assertEquals(i, openInput.readInt());
-      }
-
-      openInput.close();
-
-    }
-    IndexInput expected = dir.openInput("d1");
-    IndexInput actual = csr.openInput("d1");
-    assertSameStreams("d1", expected, actual);
-    assertSameSeekBehavior("d1", expected, actual);
-    expected.close();
-    actual.close();
-    csr.close();
-    newDir.close();
-  }
-  
-  public void testAppendTwice() throws IOException {
-    Directory newDir = newDirectory();
-    CompoundFileDirectory csw = newDir.createCompoundOutput("d.cfs");
-    createSequenceFile(newDir, "d1", (byte) 0, 15);
-    IndexOutput out = csw.createOutput("d.xyz");
-    out.writeInt(0);
-    try {
-      newDir.copy(csw, "d1", "d1");
-      fail("file does already exist");
-    } catch (IOException e) {
-      //
-    }
-    out.close();
-    assertEquals(1, csw.listAll().length);
-    assertEquals("d.xyz", csw.listAll()[0]);
-   
-    csw.close();
-
-    CompoundFileDirectory cfr = newDir.openCompoundInput("d.cfs", 1024);
-    assertEquals(1, cfr.listAll().length);
-    assertEquals("d.xyz", cfr.listAll()[0]);
-    cfr.close();
-    newDir.close();
-  }
-  
-  public void testEmptyCFS() throws IOException {
-    Directory newDir = newDirectory();
-    CompoundFileDirectory csw = newDir.createCompoundOutput("d.cfs");
-    csw.close();
-
-    CompoundFileDirectory csr = newDir.openCompoundInput("d.cfs", 1024);
-    assertEquals(0, csr.listAll().length);
-    csr.close();
-
-    newDir.close();
-  }
 }

Modified: lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java Mon Aug 22 12:07:31 2011
@@ -18,7 +18,6 @@ package org.apache.lucene.index;
  */
 
 import org.apache.lucene.util.LuceneTestCase;
-import org.apache.lucene.store.CompoundFileDirectory;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.IndexInput;
 import org.apache.lucene.store.IndexOutput;
@@ -94,7 +93,7 @@ public class TestIndexFileDeleter extend
     // figure out which field number corresponds to
     // "content", and then set our expected file names below
     // accordingly:
-    CompoundFileDirectory cfsReader = dir.openCompoundInput("_2.cfs", 1024);
+    CompoundFileReader cfsReader = new CompoundFileReader(dir, "_2.cfs");
     FieldInfos fieldInfos = new FieldInfos(cfsReader, "_2.fnm");
     int contentFieldIndex = -1;
     for(i=0;i<fieldInfos.size();i++) {

Modified: lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/store/TestFileSwitchDirectory.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/store/TestFileSwitchDirectory.java?rev=1160229&r1=1160228&r2=1160229&view=diff
==============================================================================
--- lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/store/TestFileSwitchDirectory.java (original)
+++ lucene/dev/branches/branch_3x/lucene/src/test/org/apache/lucene/store/TestFileSwitchDirectory.java Mon Aug 22 12:07:31 2011
@@ -112,44 +112,4 @@ public class TestFileSwitchDirectory ext
       dir.close();
     }
   }
-  
-  // LUCENE-3380 test that delegate compound files correctly.
-  public void testCompoundFileAppendTwice() throws IOException {
-    Directory newDir = newFSSwitchDirectory(Collections.singleton("cfs"));
-    CompoundFileDirectory csw = newDir.createCompoundOutput("d.cfs");
-    createSequenceFile(newDir, "d1", (byte) 0, 15);
-    IndexOutput out = csw.createOutput("d.xyz");
-    out.writeInt(0);
-    try {
-      newDir.copy(csw, "d1", "d1");
-      fail("file does already exist");
-    } catch (IOException e) {
-      //
-    }
-    out.close();
-    assertEquals(1, csw.listAll().length);
-    assertEquals("d.xyz", csw.listAll()[0]);
-   
-    csw.close();
-
-    CompoundFileDirectory cfr = newDir.openCompoundInput("d.cfs", 1024);
-    assertEquals(1, cfr.listAll().length);
-    assertEquals("d.xyz", cfr.listAll()[0]);
-    cfr.close();
-    newDir.close();
-  }
-  
-  /** Creates a file of the specified size with sequential data. The first
-   *  byte is written as the start byte provided. All subsequent bytes are
-   *  computed as start + offset where offset is the number of the byte.
-   */
-  private void createSequenceFile(Directory dir, String name, byte start, int size) throws IOException {
-      IndexOutput os = dir.createOutput(name);
-      for (int i=0; i < size; i++) {
-          os.writeByte(start);
-          start ++;
-      }
-      os.close();
-  }
-
 }



Mime
View raw message