lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ChristophD <christoph.die...@kqv.de>
Subject Re: Using Luke on a Lucene Index in a Database
Date Tue, 09 Jun 2009 12:47:21 GMT

Upon a request on the experiences on this issue, I am posting the most
important functions of the program. Every DB record maps directly to one
file. The function that I did not include is "getDataSource()" which
acquires a jdbc datasource to your database.

cheers,
Christoph

	private void run() throws Exception {
		DataSource ds = getDataSource();
		File toDir = new File("outputDir");
		toDir.mkdirs();
		assert toDir.exists();
		assert toDir.isDirectory();

		copyToFilesystem(ds, toDir);
	}

	public void copyToFilesystem(DataSource ds, File toDir)
			throws SQLException, IOException {
		Connection conn = ds.getConnection();
		Statement stmt = conn.createStatement();
		ResultSet rs = stmt
				.executeQuery("select NAME_, VALUE_ from XXXX.XXXX");
		while (rs.next()) {
			String name = rs.getString("NAME_");
			log.info("filename: '" + name + "'");
			InputStream inStream = rs.getBinaryStream("VALUE_");
			File file = new File(toDir, name);
			assert !file.exists();
			boolean ok = file.createNewFile();
			assert ok;
			FileOutputStream outStream = new FileOutputStream(file);

			copyLarge(inStream, outStream);
			inStream.close();
			outStream.close();
		}
		conn.rollback();
	}

	/**
	 * Taken from commons IO
	 * 
	 * @see http://svn.apache.org/viewvc/commons/proper/io/trunk/src/java/org/
	 *      apache/commons/io/IOUtils.java?revision=736890
	 */
	public static long copyLarge(InputStream input, OutputStream output)
			throws IOException {
		byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
		long count = 0;
		int n = 0;
		while (-1 != (n = input.read(buffer))) {
			output.write(buffer, 0, n);
			count += n;
		}
		return count;
	}




ChristophD wrote:
> 
> Ok, so let me clear it up.
> 
> Lucene offers different types of Directories
> (org.apache.lucene.store.Directory) into which it stores the index data.
> Most people probably use the FSDirectory implementation which writes the
> index data as files into the filesystem. However, we use the DbDirectory
> implementation which writes into a specified relational database.
> 
> Now, I was really surprised to see that Luke only offers to open an index
> that was written to the filesystem. I had expected to be able to supply a
> jdbc url.
> 
> The way Lucene writes the index into the DB is really a direct projection
> of the FS version. For every file it creates a record in the DB which has
> a name and a BLOB that contains the file's data.
> 
> So what I did was, I wrote a small program that reads the index from the
> DB and writes it back as files. These files I could open with Luke without
> problems.
> 
> so long,
> Christoph
> 
> 
> Erick Erickson wrote:
>> 
>> Well, you haven't really provided much in the way of details.For
>> instance,
>> what does it mean that your Lucene index is
>> stored in a database"? Did you store it as a BLOB? Your
>> problem statement is very hard to understand, please explain
>> in more detail. Pretend you don't know a thing about your
>> app (as in, you're just a random reader of this list) and imagine
>> you were trying to understand well enough to offer useful
>> responses...
>> 
>> Best
>> Erick
>> 
>> On Tue, May 19, 2009 at 6:12 AM, ChristophD <christoph.dietze@kqv.de>
>> wrote:
>> 
>>>
>>> This isn't really addressing my problem. I already have a running search
>>> system and just want to analyze it.
>>>
>>> cheers,
>>> Christoph
>>>
>>>
>>>
>>> amin1977 wrote:
>>> >
>>> > Are you using an object relational mapping tool like Hibernate?  if
>>> you
>>> > are
>>> > you could use hibernate search to index your persistent entities and
>>> then
>>> > use luke to inspect the indexes.   There may other ways of doing it I
>>> > guess.  Just a thought.
>>> >
>>> >
>>> > Cheers
>>> > Amin
>>> >
>>> > On Tue, May 19, 2009 at 9:23 AM, ChristophD <christoph.dietze@kqv.de>
>>> > wrote:
>>> >
>>> >>
>>> >> Hello,
>>> >>
>>> >> I would like to use Luke to connect to an existing Lucene Index which
>>> is
>>> >> stored in a Database.
>>> >> However, Luke seems only to be able to read file based indexes.
>>> >>
>>> >> What are my options to analyze the DB index?
>>> >>
>>> >> thx,
>>> >> Christoph
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://www.nabble.com/Using-Luke-on-a-Lucene-Index-in-a-Database-tp23611846p23611846.html
>>> >> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>
>>> >>
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Using-Luke-on-a-Lucene-Index-in-a-Database-tp23611846p23613338.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Using-Luke-on-a-Lucene-Index-in-a-Database-tp23611846p23942165.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message