lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dror Matalon <d...@zapatec.com>
Subject Re: Applet, read-only mode index/code contribution
Date Thu, 06 Nov 2003 18:59:08 GMT
This is also a problem when runnning a security manager on a servlet. I 
would think that you should be able to search the index without requiring
WRITE permissions on the directory.

On Thu, Nov 06, 2003 at 07:55:19PM +0100, Thomas Fuchs wrote:
> Hi,
> 
> I used lucene 1.2/it.unige.csita.lucene.RODirectory inside an applet on CD-ROM. In lucene
1.3 the system property 'disableLuceneLocks' was introduced to make it.unige.csita.lucene.RODirectory
or something like that obsolete. But this didn't work - as already mentioned by Jon - because
the security manager denies access to system properties. But lucene 1.3 rc2 and it.unige.csita.lucene.RODirectory
aren't compatible, so a work-around is still required. I attched a dirty hack based on FSDirectory,
called RODirectory.
> 
> - Thomas
> 
> > Regarding lock files created while searching, it looks like it is
> created via FSDirectory.makeLock method, which makes use of the
> disableLuceneLocks property, so this actually should take care of the
> searching locks.  So this should work on CD-ROMs.
> 
> > It would be lovely if somebody could confirm this and send a message to
> let us all know.
> 
> > Otis
> 
> >> I've been investigating using Lucene to search html pages on CD-ROM, using
> an applet as the search interface. It isn't as easy as I'd hoped...
> 
> >> One problem is that lock files are written while searching to prevent
> concurrent updates of the index -- clearly that doesn't work too well on
> read-only media! But I notice there are current changes which will make the
> use of lock files configurable.
> 
> >> Another problem is security restrictions on applets. Unfortunately one of
> these is not being able to read system properties and the new mechanism for
> turning off lock files uses a system property, so that ain't going to work
> well in my case...
> 
> >> So I've been looking at plans B, C, D, ... These would mean writing some new
> classes to specialise the existing Lucene classes to my particular case (eg
> a version of IndexReader that doesn't use a lock). But I'm not making much
> progress here since I can't extend classes because they are final or because
> they have package access.
> 
> >> Writing an application to do the same job was straightforward and gives a
> good product, so I feel I should persevere. But I do feel that everything I
> try goes across the grain of existing code.
> 
> >> Anyone got any thoughts? I would have thought that the scenario (html on
> CD-ROM + applet search) was a natural one for Lucene.  I've searched the
> archives and found several people trying to do it, but not much in the way
> of solutions.
> 
> >> Jon
> 
> ===== RODirectory.java ================================================
> 
> package ...;
> 
> /* ====================================================================
>  * The Apache Software License, Version 1.1
>  *
>  * Copyright (c) 2001, 2003 The Apache Software Foundation.  All
>  * rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>  * are met:
>  *
>  * 1. Redistributions of source code must retain the above copyright
>  *    notice, this list of conditions and the following disclaimer.
>  *
>  * 2. Redistributions in binary form must reproduce the above copyright
>  *    notice, this list of conditions and the following disclaimer in
>  *    the documentation and/or other materials provided with the
>  *    distribution.
>  *
>  * 3. The end-user documentation included with the redistribution,
>  *    if any, must include the following acknowledgment:
>  *       "This product includes software developed by the
>  *        Apache Software Foundation (http://www.apache.org/)."
>  *    Alternately, this acknowledgment may appear in the software itself,
>  *    if and wherever such third-party acknowledgments normally appear.
>  *
>  * 4. The names "Apache" and "Apache Software Foundation" and
>  *    "Apache Lucene" must not be used to endorse or promote products
>  *    derived from this software without prior written permission. For
>  *    written permission, please contact apache@apache.org.
>  *
>  * 5. Products derived from this software may not be called "Apache",
>  *    "Apache Lucene", nor may "Apache" appear in their name, without
>  *    prior written permission of the Apache Software Foundation.
>  *
>  * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
>  * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
>  * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
>  * DISCLAIMED.  IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION OR
>  * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
>  * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
>  * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
>  * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
>  * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
>  * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
>  * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>  * SUCH DAMAGE.
>  * ====================================================================
>  *
>  * This software consists of voluntary contributions made by many
>  * individuals on behalf of the Apache Software Foundation.  For more
>  * information on the Apache Software Foundation, please see
>  * <http://www.apache.org/>.
>  */
> 
> import java.io.IOException;
> import java.io.File;
> import java.io.RandomAccessFile;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.util.Hashtable;
> import java.security.MessageDigest;
> import java.security.NoSuchAlgorithmException;
> 
> import org.apache.lucene.store.*;
> import org.apache.lucene.util.Constants;
> 
> /**
>  * Straightforward implementation of {@link Directory} as a directory of files.
>  * <p>If the system property 'disableLuceneLocks' has the String value of
>  * "true", lock creation will be disabled.
>  *
>  * @see Directory
>  * @author Doug Cutting
>  */
> public final class RODirectory extends Directory {
>   /** This cache of directories ensures that there is a unique Directory
>    * instance per path, so that synchronization on the Directory can be used to
>    * synchronize access between readers and writers.
>    *
>    * This should be a WeakHashMap, so that entries can be GC'd, but that would
>    * require Java 1.2.  Instead we use refcounts...
>    */
>   private static final Hashtable DIRECTORIES = new Hashtable();
> 
>   private static final boolean DISABLE_LOCKS = true;
> 
>   private static MessageDigest DIGESTER;
>  
>   static {
>     try {
>       DIGESTER = MessageDigest.getInstance("MD5");
>     } catch (NoSuchAlgorithmException e) {
>         throw new RuntimeException(e.toString());
>     }
>   }
> 
>   /** A buffer optionally used in renameTo method */
>   private byte[] buffer = null;
> 
>   /** Returns the directory instance for the named location.
>    *
>    * <p>Directories are cached, so that, for a given canonical path, the same
>    * RODirectory instance will always be returned.  This permits
>    * synchronization on directories.
>    *
>    * @param path the path to the directory.
>    * @param create if true, create, or erase any existing contents.
>    * @return the RODirectory for the named file.  */
>   public static RODirectory getDirectory(String path, boolean create)
>       throws IOException {
>     return getDirectory(new File(path), create);
>   }
> 
>   /** Returns the directory instance for the named location.
>    *
>    * <p>Directories are cached, so that, for a given canonical path, the same
>    * RODirectory instance will always be returned.  This permits
>    * synchronization on directories.
>    *
>    * @param file the path to the directory.
>    * @param create if true, create, or erase any existing contents.
>    * @return the RODirectory for the named file.  */
>   public static RODirectory getDirectory(File file, boolean create)
>     throws IOException {
>     file = new File(file.getCanonicalPath());
>     RODirectory dir;
>     synchronized (DIRECTORIES) {
>       dir = (RODirectory)DIRECTORIES.get(file);
>       if (dir == null) {
>         dir = new RODirectory(file, create);
>         DIRECTORIES.put(file, dir);
>       } else if (create) {
>         dir.create();
>       }
>     }
>     synchronized (dir) {
>       dir.refCount++;
>     }
>     return dir;
>   }
> 
>   private File directory = null;
>   private int refCount;
> 
>   private RODirectory(File path, boolean create) throws IOException {
>     directory = path;
> 
>     if (create)
>       create();
> 
>     if (!directory.isDirectory())
>       throw new IOException(path + " not a directory");
>   }
> 
>   private synchronized void create() throws IOException {
>     if (DISABLE_LOCKS)
> 		return;
> 	if (!directory.exists())
>       if (!directory.mkdir())
>         throw new IOException("Cannot create directory: " + directory);
> 
>     String[] files = directory.list();            // clear old files
>     for (int i = 0; i < files.length; i++) {
>       File file = new File(directory, files[i]);
>       if (!file.delete())
>         throw new IOException("couldn't delete " + files[i]);
>     }
>     
>     String lockPrefix = getLockPrefix().toString(); // clear old locks
>     File tmpdir = new File(System.getProperty("java.io.tmpdir"));
>     files = tmpdir.list();
>     for (int i = 0; i < files.length; i++) {      
>       if (!files[i].startsWith(lockPrefix))
>         continue;
>       File file = new File(tmpdir, files[i]);
>       if (!file.delete())
>         throw new IOException("couldn't delete " + files[i]);
>     }
>   }
> 
>   /** Returns an array of strings, one for each file in the directory. */
>   public final String[] list() throws IOException {
>     return directory.list();
>   }
> 
>   /** Returns true iff a file with the given name exists. */
>   public final boolean fileExists(String name) throws IOException {
>     File file = new File(directory, name);
>     return file.exists();
>   }
> 
>   /** Returns the time the named file was last modified. */
>   public final long fileModified(String name) throws IOException {
>     File file = new File(directory, name);
>     return file.lastModified();
>   }
> 
>   /** Returns the time the named file was last modified. */
>   public static final long fileModified(File directory, String name)
>        throws IOException {
>     File file = new File(directory, name);
>     return file.lastModified();
>   }
> 
>   /** Set the modified time of an existing file to now. */
>   public void touchFile(String name) throws IOException {
>     File file = new File(directory, name);
>     file.setLastModified(System.currentTimeMillis());
>   }
> 
>   /** Returns the length in bytes of a file in the directory. */
>   public final long fileLength(String name) throws IOException {
>     File file = new File(directory, name);
>     return file.length();
>   }
> 
>   /** Removes an existing file in the directory. */
>   public final void deleteFile(String name) throws IOException {
>     File file = new File(directory, name);
>     if (!file.delete())
>       throw new IOException("couldn't delete " + name);
>   }
> 
>   /** Renames an existing file in the directory. */
>   public final synchronized void renameFile(String from, String to)
>       throws IOException {
>     File old = new File(directory, from);
>     File nu = new File(directory, to);
> 
>     /* This is not atomic.  If the program crashes between the call to
>        delete() and the call to renameTo() then we're screwed, but I've
>        been unable to figure out how else to do this... */
> 
>     if (nu.exists())
>       if (!nu.delete())
>         throw new IOException("couldn't delete " + to);
> 
>     // Rename the old file to the new one. Unfortunately, the renameTo()
>     // method does not work reliably under some JVMs.  Therefore, if the
>     // rename fails, we manually rename by copying the old file to the new one
>     if (!old.renameTo(nu)) {
>       java.io.InputStream in = null;
>       java.io.OutputStream out = null;
>       try {
>         in = new FileInputStream(old);
>         out = new FileOutputStream(nu);
>         // see if the buffer needs to be initialized. Initialization is
>         // only done on-demand since many VM's will never run into the renameTo
>         // bug and hence shouldn't waste 1K of mem for no reason.
>         if (buffer == null) {
>           buffer = new byte[1024];
>         }
>         int len;
>         while ((len = in.read(buffer)) >= 0) {
>           out.write(buffer, 0, len);
>         }
> 
>         // delete the old file.
>         old.delete();
>       }
>       catch (IOException ioe) {
>         throw new IOException("couldn't rename " + from + " to " + to);
>       }
>       finally {
>         if (in != null) {
>           try {
>             in.close();
>           } catch (IOException e) {
>             throw new RuntimeException("could not close input stream: " + e.getMessage());
>           }
>         }
>         if (out != null) {
>           try {
>             out.close();
>           } catch (IOException e) {
>             throw new RuntimeException("could not close output stream: " + e.getMessage());
>           }
>         }
>       }
>     }
>   }
> 
>   /** Creates a new, empty file in the directory with the given name.
>       Returns a stream writing this file. */
>   public final OutputStream createFile(String name) throws IOException {
>     return new FSOutputStream(new File(directory, name));
>   }
> 
>   /** Returns a stream reading an existing file. */
>   public final InputStream openFile(String name) throws IOException {
>     return new FSInputStream(new File(directory, name));
>   }
> 
>   /**
>    * So we can do some byte-to-hexchar conversion below
>    */
>   private static final char[] HEX_DIGITS =
>   {'0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'};
> 
>   /** Constructs a {@link Lock} with the specified name.  Locks are implemented
>    * with {@link File#createNewFile() }.
>    *
>    * <p>In JDK 1.1 or if system property <I>disableLuceneLocks</I>
is the
>    * string "true", locks are disabled.  Assigning this property any other
>    * string will <B>not</B> prevent creation of lock files.  This is useful
for
>    * using Lucene on read-only medium, such as CD-ROM.
>    *
>    * @param name the name of the lock file
>    * @return an instance of <code>Lock</code> holding the lock
>    */
>   public final Lock makeLock(String name) {
>     StringBuffer buf = getLockPrefix();
>     buf.append("-");
>     buf.append(name);
> 
>     // make the lock file in tmp, where anyone can create files.
>     final File lockFile;
> 	if (DISABLE_LOCKS)
> 		lockFile = null;
> 	else
> 		lockFile = new File(System.getProperty("java.io.tmpdir"), buf.toString());
> 
>     return new Lock() {
>       public boolean obtain() throws IOException {
>         if (DISABLE_LOCKS)
>           return true;
>         return lockFile.createNewFile();
>       }
>       public void release() {
>         if (DISABLE_LOCKS)
>           return;
>         lockFile.delete();
>       }
>       public boolean isLocked() {
>         if (DISABLE_LOCKS)
>           return false;
>         return lockFile.exists();
>       }
> 
>       public String toString() {
>         return "Lock@" + lockFile;
>       }
>     };
>   }
> 
>   private StringBuffer getLockPrefix() {
>     String dirName;                               // name to be hashed
>     try {
>       dirName = directory.getCanonicalPath();
>     } catch (IOException e) {
>       throw new RuntimeException(e.toString());
>     }
>     
>     byte digest[];
>     synchronized (DIGESTER) {
>       digest = DIGESTER.digest(dirName.getBytes());
>     }
>     StringBuffer buf = new StringBuffer();
>     buf.append("lucene-");
>     for (int i = 0; i < digest.length; i++) {
>       int b = digest[i];
>       buf.append(HEX_DIGITS[(b >> 4) & 0xf]);
>       buf.append(HEX_DIGITS[b & 0xf]);
>     }
> 
>     return buf;
>   }
> 
>   /** Closes the store to future operations. */
>   public final synchronized void close() throws IOException {
>     if (--refCount <= 0) {
>       synchronized (DIRECTORIES) {
>         DIRECTORIES.remove(directory);
>       }
>     }
>   }
> 
>   /** For debug output. */
>   public String toString() {
>     return "RODirectory@" + directory;
>   }
> }
> 
> 
> final class FSInputStream extends InputStream {
>   private class Descriptor extends RandomAccessFile {
>     /* DEBUG */
>     //private String name;
>     /* DEBUG */
>     public long position;
>     public Descriptor(File file, String mode) throws IOException {
>       super(file, mode);
>       /* DEBUG */
>       //name = file.toString();
>       //debug_printInfo("OPEN");
>       /* DEBUG */
>     }
>     
>     /* DEBUG */
>     //public void close() throws IOException {
>     //  debug_printInfo("CLOSE");
>     //    super.close();
>     //}
>     //
>     //private void debug_printInfo(String op) {
>     //  try { throw new Exception(op + " <" + name + ">"); 
>     //  } catch (Exception e) {
>     //    java.io.StringWriter sw = new java.io.StringWriter();
>     //    java.io.PrintWriter pw = new java.io.PrintWriter(sw);
>     //    e.printStackTrace(pw);
>     //    System.out.println(sw.getBuffer().toString());
>     //  }
>     //}
>     /* DEBUG */
>   }
> 
>   Descriptor file = null;
>   boolean isClone;
> 
>   public FSInputStream(File path) throws IOException {
>     file = new Descriptor(path, "r");
>     length = file.length();
>   }
> 
>   /** InputStream methods */
>   protected final void readInternal(byte[] b, int offset, int len)
>        throws IOException {
>     synchronized (file) {
>       long position = getFilePointer();
>       if (position != file.position) {
>         file.seek(position);
>         file.position = position;
>       }
>       int total = 0;
>       do {
>         int i = file.read(b, offset+total, len-total);
>         if (i == -1)
>           throw new IOException("read past EOF");
>         file.position += i;
>         total += i;
>       } while (total < len);
>     }
>   }
> 
>   public final void close() throws IOException {
>     if (!isClone)
>       file.close();
>   }
> 
>   /** Random-access methods */
>   protected final void seekInternal(long position) throws IOException {
>   }
> 
>   protected final void finalize() throws IOException {
>     close();            // close the file
>   }
> 
>   public Object clone() {
>     FSInputStream clone = (FSInputStream)super.clone();
>     clone.isClone = true;
>     return clone;
>   }
>   
>   /** Method used for testing. Returns true if the underlying
>    *  file descriptor is valid.
>    */
>   boolean isFDValid() throws IOException {
>     return file.getFD().valid();
>   }
> }
> 
> 
> final class FSOutputStream extends OutputStream {
>   RandomAccessFile file = null;
> 
>   public FSOutputStream(File path) throws IOException {
>     file = new RandomAccessFile(path, "rw");
>   }
> 
>   /** output methods: */
>   public final void flushBuffer(byte[] b, int size) throws IOException {
>     file.write(b, 0, size);
>   }
>   public final void close() throws IOException {
>     super.close();
>     file.close();
>   }
> 
>   /** Random-access methods */
>   public final void seek(long pos) throws IOException {
>     super.seek(pos);
>     file.seek(pos);
>   }
>   public final long length() throws IOException {
>     return file.length();
>   }
> 
>   protected final void finalize() throws IOException {
>     file.close();          // close the file
>   }
> 
> }
> 
> ===== RODirectory.java ================================================
> 
> ______________________________________________________________________________
> Horoskop, Comics, VIPs, Wetter, Sport und Lotto im WEB.DE Screensaver1.2
> Kostenlos downloaden: http://screensaver.web.de/?mc=021110
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 

-- 
Dror Matalon
Zapatec Inc 
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message