lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Custom Directory and IndexInput
Date Tue, 21 Aug 2012 09:18:34 GMT
I don’t understand what you are doing! Do you wrap MockIndexInputWrapper again with your
wrapper? This can of course not work, as MockIndexInputWrapper is an internal test-only class,
so the big question is:

 

What are you trying to do?

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

 <http://www.thetaphi.de/> http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: Danil ŢORIN [mailto:torindan@gmail.com] 
Sent: Tuesday, August 21, 2012 11:13 AM
To: dev@lucene.apache.org
Subject: Re: Custom Directory and IndexInput

 

Strange thing:

If I wrap all "reasonable" directories into my directory, my tests pass.

However if I wrap MockIndexInputWrapper, it calls "checkIndex()" after close on delegated
DataInput, and obviously fails to read proper data.

 

CheckIndex failed

ERROR: could not read any segments file in directory

org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource:
MockIndexInputWrapper(RAMInputStream(name=segments.gen))): -495571168 (needs to be between
-2 and -2)

          at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692)

          at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:601)

          at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:327)

          at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:350)

          at org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:192)

          at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:580)

 

Is there a clean way to force checkIndex on the top-level Directory involved in tests, not
on one of the wrappers below?

 

On Tue, Aug 21, 2012 at 11:26 AM, Uwe Schindler <uwe@thetaphi.de> wrote:

In my commit I also solved the casting problem with covariant override. But DataInput.clone()
was always public.

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/> 

eMail: uwe@thetaphi.de

 

From: Danil ŢORIN [mailto:torindan@gmail.com] 
Sent: Tuesday, August 21, 2012 9:55 AM


To: dev@lucene.apache.org
Subject: Re: Custom Directory and IndexInput

 

My bad, few casts solved the issue.

On Tue, Aug 21, 2012 at 10:46 AM, Danil ŢORIN <torindan@gmail.com> wrote:

One tricky thing, I'm doing a lot of delegation, I can't call "clone()" on the DataInput I'm
delegating to.

(unfortunately in my case it's delegation, not inheritance)

 

The fact that the delegate implements Clonable, doesn't help, the clone() method is still
protected :(.

Any idea how to circumvent this?

 

On Tue, Aug 21, 2012 at 10:37 AM, Uwe Schindler <uwe@thetaphi.de> wrote:

Thanks for „reporting“ lack of documentation, I will shortly commit some additional specs
to IndexInput’s javadocs!

 

Thanks,

Uwe

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/> 

eMail: uwe@thetaphi.de

 

From: Danil ŢORIN [mailto:torindan@gmail.com] 
Sent: Tuesday, August 21, 2012 8:59 AM
To: dev@lucene.apache.org
Subject: Re: Custom Directory and IndexInput

 

Thanks, that's exactly the response I was looking for.

On Tue, Aug 21, 2012 at 9:31 AM, Uwe Schindler <uwe@thetaphi.de> wrote:

Hi,

 

It is required that every thread uses its own IndexInput. This is why e.g. every search calls
IndexInput.clone() several times while executing a search query. Because of that there are
explicietly no thread safety requirements in IndexInput, but clone() must be implemented as
a cheap method, cloning the IndexInput and keeping the state. It is not required to recreate
file descriptors or like that, but it must make sure that the new IndexInput can be used independent,
although on same file/same file descriptor like another one.

 

Uwe

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de <http://www.thetaphi.de/> 

eMail: uwe@thetaphi.de

 

From: Danil ŢORIN [mailto:torindan@gmail.com] 
Sent: Tuesday, August 21, 2012 8:22 AM
To: java-dev@lucene.apache.org
Subject: Custom Directory and IndexInput

 


I'm trying to build a custom directory implementation, actually the directory itself just
fallback to one of the existing directory implementations, so it's actually more about IndexInput/IndexOutput.

I have some concerns about my implementation, especially related to multithreading:

*	IndexOutput writeByte/writeBytes should be no problem since only one thread will write to
specific output at any time
*	IndexInput on the other hand could be called from multiple threads
*	most implementations of IndexInput (including mine) keep some internal state like current
position in the file/buffer/etc
*	if one of the threads is calling readShort()/readLong()/readVInt() (default implementations
from DataInput) while some other thread is calling readByte() bad stuff will happen
*	still in most existing implementations I don't see any synchronization code on reads
*	is this problem solved somewhere on a higher level, like IndexReader/IndexSearcher, so I
don't have to worry about it?
*	cause I'm really not sure how to solve it without forcing synchronized on all read methods
and I'd really hate that.

How this issue is solved in Lucene? what should I do to make sure that my implementation doing
right thing?

 

Currently I'm on Lucene 4.0.0 BETA

 

 

 

 

 


Mime
View raw message