Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of cdoronc@gmail.com designates
 209.85.215.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALfq-2S86g_Ossw0MPP8N_aqAMVC7kvVFpa1vnTH15MXB0+yhw@mail.gmail.com>
References: 
 <CAJ-pxHaaOBRzQBUSFjr7=mGoK+VZbwZuGDL+OvPN4YV0=YrdzA@mail.gmail.com>
	<CAJ-pxHaO-t5xDD9=bEDh+SZtsoWF8CMcA-YcpvEYtXcgM4CNqg@mail.gmail.com>
	<CALfq-2SdmsvedBs3QcaYwnYt2UoSBwY7RRD+KGP5TJVig45=Vg@mail.gmail.com>
	<65766306-a18a-46be-8c23-db18ddb5673c@email.android.com>
	<CAJ-pxHafy=jiqEpijmxgh3nnccuANSxm3aBBVZuRZozWkvB1Ww@mail.gmail.com>
	<CALfq-2S86g_Ossw0MPP8N_aqAMVC7kvVFpa1vnTH15MXB0+yhw@mail.gmail.com>
Date: Tue, 4 Oct 2011 10:17:40 +0200
Message-ID: 
 <CAGFWK3UBLEFqQjm309o-VGBP2Y_NWXPueMisMe0v4teFkgpjtQ@mail.gmail.com>
Subject: Re: TaxWriter leakage?
From: Doron Cohen <cdoronc@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0015174beab698af6d04ae74b95f

--0015174beab698af6d04ae74b95f
Content-Type: text/plain; charset=ISO-8859-1

Looking into this with Shai I think we see how this can happen, in this code
of LTW:

  private ParentArray getParentArray() throws IOException {
    if (parentArray==null) { // [1]
      if (reader == null) {
        reader = openReader();
      }
      parentArray = new ParentArray(); // [2]
      parentArray.refresh(reader); // [3]
    }
    return parentArray;
  }

The method getParentArray() is not synchronized, and hence if there are two
calls from two different threads T1, T2, (both adding documents or such):
* T1 gets to point [1] above and see null
* T1 gets to point [2] and assigns a fresh parentArray
* Context switch, T2 gets to "drive"
* T2 gets to point [1], sees a non null parent array, and returns it
immediately

Since T1 did not yet get to point [3], the parent array is not yet
initialized, hence the NPE that T2 gets when accessing the internal array of
the parent array.

If this is the case then it is surprising that this error did not yet
happen.
Need to open a JIRA issue for this and add some synchronization.
Best if we can create this is in a standalone test.

I also think that there is nothing special in the second restart, except
that that by that time there were other servlets up (?) which were able to
trigger simultaneous AddDoc requests, exposing this bug...

Makes sense?

On Tue, Oct 4, 2011 at 6:58 AM, Shai Erera <serera@gmail.com> wrote:

> The source code is available and you can check it out from
>
> http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/lucene/contrib/facet
> .
> After you modify the code, you can call "ant jar" (if you have Ant
> installed) and it will generate the facet .jar for you. If it doesn't work
> for you, let me know and I'll generate one for you.
>
> Shai
>
> On Mon, Oct 3, 2011 at 7:21 PM, Mihai Caraman <caraman.mihai@gmail.com
> >wrote:
>
> >  Uwe :
> >
> > > Maybe another Java7 bug? Are you using Java 7?
> > >
> > Nop, java 1.6
> >
> > Shai
> > > return getParentArray().getArray()[ordinal];
> > >
> > Can you give me a lucene jar with print outs for when it throws this NPE?
> >
> > ...How is using ThreadedIndexWriter  related
> > >
> > When I remove the Threaded version, it doesn't give me any problems.
> > Otherwise, when i first deploy the jar, (and maybe because it gets a bit
> > overwhelmed with accumulated data(from a distributed queue-Kestrel) it
> > throws that nasty error for every Indexing thread, in my case for all 6
> of.
> > After that if i restart the server, no more exception.
> >
> > I tried to reproduce the error, with only the Readers and writers and
> with
> > some bogus data, but it didn't throw anything.
> >
>

--0015174beab698af6d04ae74b95f--