Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of naamakraus@gmail.com
 designates 209.85.222.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=KDlhW1ojVpdtHc7Hyie4R+3RhL8mKk/HFkMXqxRXYy+NPzxL74soRqG2RDqdqZx8kG
         PgPbeGT9KfzVEQLHfGoCOKR3IF6NwFzkXRkD/ud/aZGgZe1/EfFy8uWYHcIOiVyylWYb
         qRQa5woS6I2g/9SbhwjxnDQgsUMW7LLVxHGzU=
MIME-Version: 1.0
In-Reply-To: <9ac0c6aa1001070857i6de5c499m9611dc979f6c6110@mail.gmail.com>
References: <643aa4871001070413x7629dc2ao5661a18880fcadc4@mail.gmail.com>
	<359a92831001070537o205957e7nd3d14a0797f5d4ab@mail.gmail.com>
	<643aa4871001070741t13a8a24coa4a92057bd4e27a7@mail.gmail.com>
	<359a92831001070851s1c9c562fjf05925af1402fa3d@mail.gmail.com>
	<9ac0c6aa1001070857i6de5c499m9611dc979f6c6110@mail.gmail.com>
From: Naama Kraus <naamakraus@gmail.com>
Date: Fri, 8 Jan 2010 00:09:02 +0200
Message-ID: <643aa4871001071409n5e7a95bdr973fa4e1e48602b8@mail.gmail.com>
Subject: Re: Problems with IndexWriter#commit() on Linux
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1145cc35668047c9a5294

--000e0cd1145cc35668047c9a5294
Content-Type: text/plain; charset=ISO-8859-1

Thanks all for the hints, I'll get back to my code and do some additional
checks.
Naama

On Thu, Jan 7, 2010 at 6:57 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> kill -9 is harsh, but, perfectly fine from Lucene's standpoint.
> Likewise if the OS or JVM crashes, power is suddenly lost, the index
> will just fallback to the last successful commit.  What will cause
> corruption is if you have bit errors happening somewhere in the
> machine... or if two writers are accidentally allowed to be open on
> one index... then you're in trouble.
>
> What IO system (filesystem & hardware) are you using on Linux?
> Boiling down to a smallish test case can help to isolate the
> problem...
>
> Mike
>
> On Thu, Jan 7, 2010 at 11:51 AM, Erick Erickson <erickerickson@gmail.com>
> wrote:
> > Can you show us the code where you commit?
> >
> > And how do you kill your process? Kill -9 is...er...harsh....
> >
> > Yeah, I'm wondering whether the index file size *stays*
> > changed after you kill you process. If it keeps its
> > growing on every run (after you kill your process
> > multiple times), then I'd suspect that you aren't
> > adding documents like you think you are. Perhaps
> > different fields, different analyzers, etc.
> >
> > Luke should show you the largest document by ID,
> > as well as document counts. Comparing changes
> > in the document count and the max doc ID should
> > tell you something...
> >
> > Is it possible that you are updating existing docs
> > rather than adding new ones?
> >
> > Best
> > Erick
> >
> > On Thu, Jan 7, 2010 at 10:41 AM, Naama Kraus <naamakraus@gmail.com>
> wrote:
> >
> >> Thanks dor the input.
> >>
> >> 1. While the process is running, I do see the index files growing on
> disk
> >> and the time stamps changing. Should I see a change in size right after
> >> killing the process, is that what you mean ?
> >> 2. Yes, same directory is being used for indexing and search.
> >> 3. Didn't try Luke, good idea. Though I wonder, the same code runs well
> on
> >> Windows.
> >>
> >> Naama
> >>
> >> On Thu, Jan 7, 2010 at 3:37 PM, Erick Erickson <erickerickson@gmail.com
> >> >wrote:
> >>
> >> > Several questions:
> >> > 1> are the index files larger after you kill your process?
> >> >    Or have the timestamps changed?
> >> > 2> are you absolutely sure that your indexer, when you
> >> >     add documents, is pointing at the same directory your
> >> >     search is pointing to?
> >> > 3> Have you gotten a copy of Luke and examined your index
> >> >     to see if, perhaps, your documents aren't being added the
> >> >     way you think they are?
> >> >
> >> > Erick
> >> >
> >> > On Thu, Jan 7, 2010 at 7:13 AM, Naama Kraus <naamakraus@gmail.com>
> >> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I am using IndexWriter#commit() methods in my program to commit
> >> document
> >> > > additions to the index. I do that once in a while, after a bunch of
> >> > > documents were added. Since my indexing process is long, I want to
> make
> >> > > sure
> >> > > I don't loose too many additions in case of a crash.
> >> > > When running on Windows, things work as expected. But when running
> my
> >> > code
> >> > > on Linux, seems like commit() has no effect. If I kill my program
> and
> >> > then
> >> > > restart it, I don't see documents that I added and then committed
> (they
> >> > are
> >> > > not returned by a search operation).
> >> > > I am running Lucene 3.0.0
> >> > >
> >> > > Can anyone help ?
> >> > >
> >> > > Thanks, Naama
> >> > >
> >> > > --
> >> > > "If you want your children to be intelligent, read them fairy tales.
> If
> >> > you
> >> > > want them to be more intelligent, read them more fairy tales."
> >> > > "What really interests me is whether God had any choice in the
> creation
> >> > of
> >> > > the world."
> >> > > (Albert Einstein)
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> "If you want your children to be intelligent, read them fairy tales. If
> you
> >> want them to be more intelligent, read them more fairy tales."
> >> "What really interests me is whether God had any choice in the creation
> of
> >> the world."
> >> (Albert Einstein)
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
"If you want your children to be intelligent, read them fairy tales. If you
want them to be more intelligent, read them more fairy tales."
"What really interests me is whether God had any choice in the creation of
the world."
(Albert Einstein)

--000e0cd1145cc35668047c9a5294--