Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of simon.willnauer@googlemail.com
 designates 209.85.212.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:reply-to:in-reply-to:references:date:message-id
         :subject:from:to:cc:content-type:content-transfer-encoding;
        b=TQaKbtbkarl2qyhjOWlMrpATRtbkI8jQhixCGnS8c54Z/++yzLBQLYDH6MqK8lSawB
         HQLMXTjjfENRvhBdBVCMKZIiX2v5FxEALmQeAjYxHATBklRCDFFKtqx60j10M13x6WZV
         9g+nR46bvh0smcT8a2Vg/6os/CAK/mP1UpTuk=
MIME-Version: 1.0
Reply-To: simon.willnauer@gmail.com
In-Reply-To: <BANLkTimJ=KsvtnN4jMWEDUx3BAmzXGNisg@mail.gmail.com>
References: <AANLkTikMY=eVoCCAoVZoQdgj-b4EuKvxXxoh1cnHY5=q@mail.gmail.com>
	<AANLkTi=zJxmB5Gt-W+FrN+QZPkhMiPYMMMy5ij=MmLAh@mail.gmail.com>
	<AANLkTi=xBrJcLvmA6QjWQgdjRXya0nZBV9GmpVwePtPJ@mail.gmail.com>
	<AANLkTi=M7z1Ztzu_L1uPocuFiGj03tKrh-QoVaMjURFb@mail.gmail.com>
	<AANLkTin-8NTb7zJgtAUwNTMtdfkA_ORwovBBGPxdwzk+@mail.gmail.com>
	<AANLkTinL2d9VqpPb2f2ANVPgx8cL1wmGK-YvXDByNNwf@mail.gmail.com>
	<BANLkTimJ=KsvtnN4jMWEDUx3BAmzXGNisg@mail.gmail.com>
Date: Thu, 31 Mar 2011 21:10:01 +0200
Message-ID: <AANLkTin6YGqgW=SWSw7WvbdT+CJiGy=Ub+kf=+JDeBF2@mail.gmail.com>
Subject: Re: a faster way to addDocument and get the ID just added?
From: Simon Willnauer <simon.willnauer@googlemail.com>
To: java-user@lucene.apache.org
Cc: Ian Lea <ian.lea@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hey Ian,

On Thu, Mar 31, 2011 at 11:32 AM, Ian Lea <ian.lea@gmail.com> wrote:
>>> Subject: a faster way to addDocument and get the ID just added?
>
> Might it be possible to come up with a version of
> IndexWriter.addDocument() that returns the docid rather than void?
> Answering that question is way out of my league, but it would
> presumably be quick.
>
With the current trunk I think we could do that since doc IDs are
assigned in DocumentsWriter and we only have one instance of this
although we are indexing into multiple mem segments and merge on
flush. But, (yeah there must be a but :) we are working on
DocumentsWriterPerThread to exploit extra concurrency for 4.0 where
this doesn't work anymore since we indexing a segment per thread which
is not merged on flush but written directly to disk.

With 4.0 we will also have Column Stride Fields which might help with
this issue which enables you to use your own docIds stored in a fast
accessible integrated column based storage. It might not be as fast as
docIds directly but reasonable since it can be access with a low
footprint iterator during scoring.

maybe that helps once 4.0 is there

simon
> --
> Ian.
>
>
> On Thu, Mar 31, 2011 at 6:34 AM, Trejkaz <trejkaz@trypticon.org> wrote:
>> On Wed, Mar 30, 2011 at 8:21 PM, Simon Willnauer
>> <simon.willnauer@googlemail.com> wrote:
>>> Before trunk (and I think
>>> its in 3.1 also) merge only merged continuous segments so the actual
>>> per-segment ID might change but the global document ID doesn't if you
>>> only add documents. But this should not be considered a feature. In
>>> upcoming version this does not work anymore since merges can now be
>>> non-continuous.
>>
>> This myth was busted some time ago:
>> https://issues.apache.org/jira/browse/LUCENE-2506?#comment-12935973
>>
>> Summary: selecting segments to merge is decided by MergePolicy, and a
>> MergePolicy which does not upset ordering will be remain in existence.
>>
>>> Anyway, I strongly discourage to rely on lucene document IDs you
>>> should not do this at all. Can't you use your own ID mechanism?
>>
>> This has pretty much already been covered in my reply to the previous
>> person that suggested that solution, not to mention in the initial
>> email which started the thread.
>>
>> Summary: the overheads are simply not acceptable.
>>
>> So far the only remotely helpful suggestion I have heard anywhere is
>> to keep two gigantic int[] arrays in memory, mapping the IDs in each
>> direction. =C2=A0This would work if we had an infinite amount of memory =
to
>> play with, but unfortunately we don't. =C2=A01 billion item indexes are
>> expected to work, and we can't just tell everyone to buy 8 GB more RAM
>> when we update to the next version of our app. =C2=A0If we were a
>> server-side app, *maybe* we could...
>>
>> TX
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org