Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
MIME-Version: 1.0
References: <CADpRcOOLf3MD5KS0tLt8-D6+iCVJGkVU35woD0hZpjpsBOY4fQ@mail.gmail.com>
 <CAPz8bx1opkgnydc_2qEah6epefy_-7t4E+ZPuLCaYdfjSsYr7Q@mail.gmail.com>
 <CADpRcOPs3b1PkDxvR_=VPi+uLUPTbdO5=ajAzFFr3jEmbdtcOQ@mail.gmail.com>
 <CAPz8bx3mAgswL_VRD2a1hTvNf_VfjbNx2yTQazujZ7Lu2TcdVg@mail.gmail.com> <CAPsWd+MOANAzo=pmsGsSiQ_1A3tivCmX+-M+_U528X8dZRrY9Q@mail.gmail.com>
In-Reply-To: <CAPsWd+MOANAzo=pmsGsSiQ_1A3tivCmX+-M+_U528X8dZRrY9Q@mail.gmail.com>
From: =?UTF-8?B?S3VkcmV0dGluIEfDvGxlcnnDvHo=?= <kudrettin@gmail.com>
Date: Tue, 09 May 2017 17:13:45 +0000
Message-ID: <CADpRcONzW80FX0RqHfjfdLcY5x31N3USCufvKOqqBq9ZhXMpMQ@mail.gmail.com>
Subject: Re: Lucene update performance
To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>
Content-Type: multipart/alternative; boundary=001a1134e2180bea04054f1a7b11
archived-at: Tue, 09 May 2017 17:14:07 -0000

--001a1134e2180bea04054f1a7b11
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Fair enough, however, I see this:
$ cat log
Tue May  9 07:19:45 EDT 2017: Indexing starts
Tue May  9 07:32:33 EDT 2017: Deletion starts with a list of 1278635 files
Tue May  9 07:49:47 EDT 2017: Deletion complete, Addition starts with
1272334 files

$ date
Tue May  9 13:12:58 EDT 2017

I am using two phase commit model. Deletion logic above utilizes
writer.deleteDocuments(query), and addition utilizes
writer.addDocument(doc). Judging simply from this log deletion doesn't seem
to be taking long. What am I missing?


On Tue, May 9, 2017 at 10:23 AM Adrien Grand <jpountz@gmail.com> wrote:

> addDocument can be a significant gain compared to updateDocument as doing=
 a
> PK lookup on a unique field has a cost that is not negligible compared to
> indexing a document, especially if the indexing chain is simple (no large
> text fields with complex analyzers). Reindexing in place will also cause
> more merging. Overall I find the 3x factor a bit high, but not too
> surprising if documents and the analysis chain are simple, and/or if
> storage is slow.
>
> Le mar. 9 mai 2017 =C3=A0 16:06, Rob Audenaerde <rob.audenaerde@gmail.com=
> a
> =C3=A9crit :
>
> > As far as I know, the updateDocument method on the IndexWriter delete a=
nd
> > add. See also the javadoc:
> >
> > [..] Updates a document by first deleting the document(s)
> >     containing term and then adding the new
> >     document.  The delete and then add are atomic as seen
> >     by a reader on the same index (flush may happen only after
> >     the add). [..]
> >
> >
> > On Tue, May 9, 2017 at 3:37 PM, Kudrettin G=C3=BClery=C3=BCz <kudrettin=
@gmail.com>
> > wrote:
> >
> > > I do update the entire document each time. Furthermore, this sometime=
s
> > > means deleting compressed archives which are stores as multiple
> documents
> > > for each compressed archive file and readding them.
> > >
> > > Is there an update method, is it better performance than remove then
> > add? I
> > > was simply removing modified files from the index (which doesn't seem
> to
> > > take long), and readd them.
> > >
> > > On Tue, May 9, 2017 at 9:33 AM Rob Audenaerde <
> rob.audenaerde@gmail.com>
> > > wrote:
> > >
> > > > Do you update each entire document? (vs updating numeric docvalues?=
)
> > > >
> > > > That is implemented as 'delete and add' so I guess that will be
> slower
> > > than
> > > > clean sheet indexing. Not sure if it is 3x slower, that seems a bit
> > much?
> > > >
> > > > On Tue, May 9, 2017 at 3:24 PM, Kudrettin G=C3=BClery=C3=BCz <
> > kudrettin@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > For a 5.2.1 index that contains around 1.2 million documents,
> > updating
> > > > the
> > > > > index with 1.3 million files seems to take 3X longer than doing a
> > > scratch
> > > > > indexing. (Files are crawled over NFS, indexes are stored on a
> > > mechanical
> > > > > disk locally (Btrfs))
> > > > >
> > > > > Is this expected for Lucene's update index logic, or should I
> further
> > > > debug
> > > > > my part of the code for update performance?
> > > > >
> > > > > Thank you,
> > > > > Kudret
> > > > >
> > > >
> > >
> >
>

--001a1134e2180bea04054f1a7b11--