Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
MIME-Version: 1.0
References: <CAMojAe_wmaXK8YO_Qhh9UVUOeZvsw0btrfm6mjPk7poYfk606w@mail.gmail.com>
 <CAL8PwkaSw5twfn8Ou_dj33_gAkwaaR0W8zPzXEmWiC6vi_m6yg@mail.gmail.com>
 <CAMojAe_mXq4LdZjH+F=w7GvtQ1zHD_ksgUgjWUofRT65ahoHaA@mail.gmail.com> <CAL8PwkaNbg34b3x6_OKnFnZyynb_wDyJOxSBYOfemUExcepFpw@mail.gmail.com>
In-Reply-To: <CAL8PwkaNbg34b3x6_OKnFnZyynb_wDyJOxSBYOfemUExcepFpw@mail.gmail.com>
From: Baskakov Daniel <gdaniq@gmail.com>
Date: Mon, 27 Jun 2016 09:07:11 +0000
Message-ID: <CAMojAe_TKLcSE19VCECfQLDaiSpbNEwgFGYtRdc8C2HTRa22jQ@mail.gmail.com>
Subject: Re: Synchronous Lucene index update tests occasionally fail
To: Lucene Users <java-user@lucene.apache.org>
Content-Type: multipart/alternative; boundary=089e0122ebe2146a7305363ed9d6
archived-at: Mon, 27 Jun 2016 09:07:30 -0000

--089e0122ebe2146a7305363ed9d6
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I've just noticed that not only dynamic adding/removing entities tests
fail, but also a simple indexing.

Here is a boiled down structure of the test:

  @BeforeClass
  public static void beforeClass() throws Exception
  {
    // ContextManager is a domain model
    contextManager =3D createContextManager();

    searcher =3D new
ServerSearcher(Collections.singletonList(indexingSettings), false);

    searcher.openRamDirectory();

    // Context is a domain model item, it has variables. One of the
contexts has "variableWithHelpString" variable.
    // searcher.createContextDocument(aContext) creates a Lucene document
with a field: new TextField(field, value, Field.Store.YES)
    for (Context aContext : contextManager)
    {
      Document doc =3D searcher.createContextDocument(aContext);

      final Term idTerm =3D new Term(ID_FIELD, doc.get(ID_FIELD));
      // using updateDocument here because the domain model is dynamic
      searcher.getIndexWriter().updateDocument(idTerm, doc);
    }
 }

  @Test
  public void testVariableName() throws Exception
  {
    searcher.commitNow();

    String text =3D "variableWithHelpString";

    final MultiFieldQueryParser queryParser =3D new
MultiFieldQueryParser(ServerSearcher.ALL_SEARCH_FIELDS, searcher.analyzer);
    queryParser.setDefaultOperator(QueryParser.Operator.AND);

    text =3D text.replaceAll("(\\w+)", "$1\\*");

    final Query query =3D queryParser.parse(text);

    final IndexSearcher searcher =3D searcher.acquireIndexSearcher();

    TopScoreDocCollector docCollector =3D TopScoreDocCollector.create(5000)=
;

    searcher.search(query, docCollector);

    final ScoreDoc[] scoreDocs =3D docCollector.topDocs().scoreDocs;

    assertThat(scoreDocs.length(), is(1));
  }

Also there is a comprehensive logging during the test invocation. And it
can be seen that document for the context with 'variableWithHelpString' is
properly created and added to IW:
12:34:23,499 DEBUG ag.context.search         Variable Definition document
added to index:
Document<stored,indexed,indexOptions=3DDOCS<id:rootcontext:variableWithHelp=
String>
stored,indexed,tokenized,omitNorms,indexOptions=3DDOCS,numericType=3DINT,nu=
mericPrecisionStep=3D8<docType:1>
stored,indexed,indexOptions=3DDOCS<contextPath:rootcontext>
stored,indexed,tokenized<name:variableWithHelpString>
stored,indexed,tokenized<description:variableWithHelpString>
stored,indexed,tokenized<value:Two Beer Or Not Two Beer?, , >
stored,indexed,tokenized<fields:Dummy Field (dummyField)>
stored,indexed,tokenized<fields:Variable Field (variableField) [Variable
Field Help]> stored,indexed,tokenized<fields:Dummy Field (dummyField1)>>
  -     -     -     -     -     -     -     -     -     -     [main]

Here is the later log output for search operation that returns no document:
12:34:23,796 DEBUG ag.context.search         Search
'name:variablewithhelpstring* type:variablewithhelpstring*
description:variablewithhelpstring* help:variablewithhelpstring*
fields:variablewithhelpstring* outputFields:variablewithhelpstring*
value:variablewithhelpstring*' took '0' seconds and returned 0 hits     -
  -     -     -     -     -     -     -     -     -     [main]

Daniel.

=D0=BF=D0=BD, 27 =D0=B8=D1=8E=D0=BD. 2016 =D0=B3. =D0=B2 11:12, Michael McC=
andless <lucene@mikemccandless.com>:

> Can you boil this down to a small standalone test case showing the issue?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Mon, Jun 27, 2016 at 4:03 AM, Baskakov Daniel <gdaniq@gmail.com> wrote=
:
>
>> Thank you Mike.
>>
>> Commit is performed after each indexing op in unit tests only:
>>
>>   public void commitNow() throws IOException
>>   {
>>     if (getIndexWriter().hasUncommittedChanges())
>>     {
>>       getIndexWriter().commit();
>>     }
>>   }
>>
>> In production environment I have a timer that performs commit periodical=
ly
>> if required.
>>
>> I do reopen near-real-time IR before every test search (thanks to your
>> blog!):
>>
>>   private IndexSearcher acquireIndexSearcher() throws IOException
>>   {
>>     if (searcherManager =3D=3D null)
>>     {
>>       searcherManager =3D new SearcherManager(getIndexWriter(), true, nu=
ll);
>>     }
>>     searcherManager.maybeRefreshBlocking();
>>     return searcherManager.acquire();
>>   }
>>
>> But the problem is still there.
>>
>> Daniel.
>>
>> =D1=87=D1=82, 23 =D0=B8=D1=8E=D0=BD. 2016 =D0=B3. =D0=B2 17:19, Michael =
McCandless <
>> lucene@mikemccandless.com>:
>>
>> > You must reopen your IndexReader to see recent changes to the index.
>> >
>> > But, IW.commit after each indexing op is very costly.
>> >
>> > It's much better to get near-real-time readers, e.g. from a
>> > SearcherManager that you pass your IW instance too, after each set of
>> > changes that you now need to search.
>> >
>> > As long as you call SearcherManager.maybeRefreshBlocking after changes
>> to
>> > the IW, the resulting reopened reader will reflect your index changes.
>> >
>> > Mike McCandless
>> >
>> > http://blog.mikemccandless.com
>> >
>> > On Thu, Jun 23, 2016 at 7:47 AM, Baskakov Daniel <gdaniq@gmail.com>
>> wrote:
>> >
>> >> Originally i've posted the question at stackoverflow.com but without
>> any
>> >> reply. So I hope someone can help me in the official list.
>> >>
>> >> I'm testing that dynamic changes of the domain model reflects at the
>> >> Lucene
>> >> index. Special event listeners (synchronous, no multithreading here)
>> are
>> >> executed when the domain model components change. Listeners update th=
e
>> >> Lucene index:
>> >>
>> >> Document doc =3D createDocumentForComponent(domainModelComponent);
>> >> indexWriter.updateDocument(docTerm, doc);
>> >> indexWriter.commit();
>> >>
>> >> Then I perform searching by a query that contains recently added
>> changes.
>> >> Most of the time tests work perfect, but sometimes they fail
>> (especially
>> >> in
>> >> automated builds).
>> >>
>> >> I've tried to acquire an IndexSearcher by different ways: create a ne=
w
>> >> searcher on the same Directory or obtain it via SearcherManager.
>> >>
>> >> Is there a way to made recent index changes available to index search=
er
>> >> with 100% confidence?
>> >>
>> >
>> >
>>
>
>

--089e0122ebe2146a7305363ed9d6--