lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danil ŢORIN <torin...@gmail.com>
Subject Re: Custom Directory and IndexInput
Date Tue, 21 Aug 2012 11:56:31 GMT
Unfortunately test framework is not flexible enough.

My constructor needs another directory, LuceneTestCase expect either a
File, either a zero-arg constructor.

On Tue, Aug 21, 2012 at 1:19 PM, Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi,****
>
> ** **
>
> OK. If you test against only Lucene Core’s directories you are fine.
> Wrapping the special wrappers from test-framework is not needed and wrong,
> they are just for testing Lucene itsself. In fact when running all Lucene
> tests against you own directory (-Dtests.directory=…), the test framework
> will take care and wrap YOUR Directory with MockDirectoryWrapper to assert
> some special things (like Lucene code closes everything correctly, Index is
> correct,…)****
>
> ** **
>
> Uwe****
>
> ** **
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ** **
>
> *From:* Danil ŢORIN [mailto:torindan@gmail.com]
> *Sent:* Tuesday, August 21, 2012 11:34 AM
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: Custom Directory and IndexInput****
>
> ** **
>
> I was trying to run the whole suite of randomized tests, wrapped inside my
> own implementation.****
>
> ** **
>
> I wasn't sure how my stuff will behave against MMap, NIO, RAM, NRT
> implementations, so I thought running fully randomized tests would work to
> find edge cases in my implementation.****
>
> ** **
>
> By randomizing everything I found MockDirectoryWrapper with this issue.***
> *
>
> ** **
>
> I'll retry my tests without MockDirectoryWrapper.****
>
> ** **
>
> On Tue, Aug 21, 2012 at 12:18 PM, Uwe Schindler <uwe@thetaphi.de> wrote:**
> **
>
> I don’t understand what you are doing! Do you wrap MockIndexInputWrapper
> again with your wrapper? This can of course not work, as
> MockIndexInputWrapper is an internal test-only class, so the big question
> is:****
>
>  ****
>
> What are you trying to do?****
>
>  ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
>  ****
>
> *From:* Danil ŢORIN [mailto:torindan@gmail.com]
> *Sent:* Tuesday, August 21, 2012 11:13 AM****
>
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: Custom Directory and IndexInput****
>
>  ****
>
> Strange thing:****
>
> If I wrap all "reasonable" directories into my directory, my tests pass.**
> **
>
> However if I wrap MockIndexInputWrapper, it calls "checkIndex()" after
> close on delegated DataInput, and obviously fails to read proper data.****
>
>  ****
>
> CheckIndex failed****
>
> ERROR: could not read any segments file in directory****
>
> org.apache.lucene.index.IndexFormatTooNewException: Format version is not
> supported (resource:
> MockIndexInputWrapper(RAMInputStream(name=segments.gen))): -495571168
> (needs to be between -2 and -2)****
>
>           at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692)
> ****
>
>           at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:601)
> ****
>
>           at
> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:327)****
>
>           at
> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:350)****
>
>           at
> org.apache.lucene.util._TestUtil.checkIndex(_TestUtil.java:192)****
>
>           at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:580)
> ****
>
>  ****
>
> Is there a clean way to force checkIndex on the top-level Directory
> involved in tests, not on one of the wrappers below?****
>
>  ****
>
> On Tue, Aug 21, 2012 at 11:26 AM, Uwe Schindler <uwe@thetaphi.de> wrote:**
> **
>
> In my commit I also solved the casting problem with covariant override.
> But DataInput.clone() was always public.****
>
>  ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
>  ****
>
> *From:* Danil ŢORIN [mailto:torindan@gmail.com]
> *Sent:* Tuesday, August 21, 2012 9:55 AM****
>
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: Custom Directory and IndexInput****
>
>  ****
>
> My bad, few casts solved the issue.****
>
> On Tue, Aug 21, 2012 at 10:46 AM, Danil ŢORIN <torindan@gmail.com> wrote:*
> ***
>
> One tricky thing, I'm doing a lot of delegation, I can't call "clone()" on
> the DataInput I'm delegating to.****
>
> (unfortunately in my case it's delegation, not inheritance)****
>
>  ****
>
> The fact that the delegate implements Clonable, doesn't help, the clone()
> method is still protected :(.****
>
> Any idea how to circumvent this?****
>
>  ****
>
> On Tue, Aug 21, 2012 at 10:37 AM, Uwe Schindler <uwe@thetaphi.de> wrote:**
> **
>
> Thanks for „reporting“ lack of documentation, I will shortly commit some
> additional specs to IndexInput’s javadocs!****
>
>  ****
>
> Thanks,****
>
> Uwe****
>
>  ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
>  ****
>
> *From:* Danil ŢORIN [mailto:torindan@gmail.com]
> *Sent:* Tuesday, August 21, 2012 8:59 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: Custom Directory and IndexInput****
>
>  ****
>
> Thanks, that's exactly the response I was looking for.****
>
> On Tue, Aug 21, 2012 at 9:31 AM, Uwe Schindler <uwe@thetaphi.de> wrote:***
> *
>
> Hi,****
>
>  ****
>
> It is required that every thread uses its own IndexInput. This is why e.g.
> every search calls IndexInput.clone() several times while executing a
> search query. Because of that there are explicietly no thread safety
> requirements in IndexInput, but clone() must be implemented as a cheap
> method, cloning the IndexInput and keeping the state. It is not required to
> recreate file descriptors or like that, but it must make sure that the new
> IndexInput can be used independent, although on same file/same file
> descriptor like another one.****
>
>  ****
>
> Uwe****
>
>  ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
>  ****
>
> *From:* Danil ŢORIN [mailto:torindan@gmail.com]
> *Sent:* Tuesday, August 21, 2012 8:22 AM
> *To:* java-dev@lucene.apache.org
> *Subject:* Custom Directory and IndexInput****
>
>  ****
>
>
> I'm trying to build a custom directory implementation, actually the
> directory itself just fallback to one of the existing directory
> implementations, so it's actually more about IndexInput/IndexOutput.
>
> I have some concerns about my implementation, especially related to
> multithreading:****
>
>    - IndexOutput writeByte/writeBytes should be no problem since only one
>    thread will write to specific output at any time****
>    - IndexInput on the other hand could be called from multiple threads***
>    *
>    - most implementations of IndexInput (including mine) keep some
>    internal state like current position in the file/buffer/etc****
>    - if one of the threads is calling readShort()/readLong()/readVInt()
>    (default implementations from DataInput) while some other thread is calling
>    readByte() bad stuff will happen****
>    - still in most existing implementations I don't see any
>    synchronization code on reads****
>    - is this problem solved somewhere on a higher level, like
>    IndexReader/IndexSearcher, so I don't have to worry about it?****
>    - cause I'm really not sure how to solve it without forcing
>    synchronized on all read methods and I'd really hate that.****
>
> How this issue is solved in Lucene? what should I do to make sure that my
> implementation doing right thing?****
>
>  ****
>
> Currently I'm on Lucene 4.0.0 BETA****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> ** **
>

Mime
View raw message