Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 21731 invoked from network); 8 Oct 2009 18:41:51 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 8 Oct 2009 18:41:51 -0000 Received: (qmail 67431 invoked by uid 500); 8 Oct 2009 18:41:51 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 67343 invoked by uid 500); 8 Oct 2009 18:41:50 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 67335 invoked by uid 99); 8 Oct 2009 18:41:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2009 18:41:50 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of john.wang@gmail.com designates 209.85.210.192 as permitted sender) Received: from [209.85.210.192] (HELO mail-yx0-f192.google.com) (209.85.210.192) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2009 18:41:39 +0000 Received: by yxe30 with SMTP id 30so8831527yxe.29 for ; Thu, 08 Oct 2009 11:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=zaABsYT74bGUdu+9WbBSTSh5PUrlu72xU0Mg8o/Rr2c=; b=k+nIZJn8A9tfX6vPYnyWpartZENaP/SGltGSlXC2vul/CN9Ccdz5isWufvmIjeg3gl wRI7JAqbaT7j2iDs7DIF5TQ78j0QtA/vrzcr+xsTEZ696+bVTsrZgYqavENQDeZ3tVhf P6AHPKWo8Hd2BtBfjaHuR6fGs1fBjS6HeYU5g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=PH8dC5YXcHS8hba6ZnxJcTiqUo5JMo9vtt7zdxPfwKDdQkdzFt1RX+/E2h1Cj3VBCi RhOYamwaun+t9Gbi8sx44dxcZHu2NoPINbF2Ojc7CfjpPRBh1np7cxwN7f1/KrOyouix Swx79famDs2O/bf41yHnN7pRH1oI23symFiUg= MIME-Version: 1.0 Received: by 10.101.96.18 with SMTP id y18mr1715122anl.174.1255027277976; Thu, 08 Oct 2009 11:41:17 -0700 (PDT) In-Reply-To: <31EC32D34DE94D449EF9A82C4CEFCBCE@VEGA> References: <1307591248.1227004424235.JavaMail.jira@brutus> <418830629.1253798056142.JavaMail.jira@brutus> <31EC32D34DE94D449EF9A82C4CEFCBCE@VEGA> Date: Thu, 8 Oct 2009 11:41:17 -0700 Message-ID: <8837fb770910081141k36cfa466y5ca9fce9feb9638e@mail.gmail.com> Subject: Re: [jira] Commented: (LUCENE-1458) Further steps towards flexible indexing From: John Wang To: java-dev@lucene.apache.org Content-Type: multipart/alternative; boundary=001636ed77c51654bc047570cf4d X-Virus-Checked: Checked by ClamAV on apache.org --001636ed77c51654bc047570cf4d Content-Type: text/plain; charset=ISO-8859-1 Hi guys: What are your thoughts about contributing Kamikaze as a lucene contrib package? We just finished porting kamikaze to lucene 2.9. With the new 2.9 api, it allows us for some more code tuning and optimization improvements. We will be releasing kamikaze, it might a good time to add it to the lucene contrib package if there is interest. Thanks -John On Thu, Sep 24, 2009 at 6:20 AM, Uwe Schindler wrote: > By the way: In the last RC of Lucene 2.9 we added a new method to DocIdSet > called isCacheable(). It is used by e.g. CachingWrapperFilter to determine, > if a DocIdSet is easy cacheable or must be copied to an OpenBitSetDISI (the > default is false, so all custom DocIdSets are copied to OpenBitSetDISI by > CachingWrapperFilter, even if not needed - if a DocIdSet does not do disk > IO > and have a fast iterator like e.g. the FieldCache ones in > FieldCacheRangeFilter, it should return true; see CHANGES.txt). Maybe this > should also be added to Kamikaze, which is a really nice project! > Especially > filter DocIdSets should pass this method to its delegate (see > FilterDocIdSet > in Lucene). > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: uwe@thetaphi.de > > > > -----Original Message----- > > From: John Wang (JIRA) [mailto:jira@apache.org] > > Sent: Thursday, September 24, 2009 3:14 PM > > To: java-dev@lucene.apache.org > > Subject: [jira] Commented: (LUCENE-1458) Further steps towards flexible > > indexing > > > > > > [ https://issues.apache.org/jira/browse/LUCENE- > > 1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment- > > tabpanel&focusedCommentId=12759112#action_12759112 ] > > > > John Wang commented on LUCENE-1458: > > ----------------------------------- > > > > Just a FYI: Kamikaze was originally started as our sandbox for Lucene > > contributions until 2.4 is ready. (we needed the DocIdSet/Iterator > > abstraction that was migrated from Solr) > > > > It has three components: > > > > 1) P4Delta > > 2) Logical boolean operations on DocIdSet/Iterators (I have created a > jira > > ticket and a patch for Lucene awhile ago with performance numbers. It is > > significantly faster than DisjunctionScorer) > > 3) algorithm to determine which DocIdSet implementations to use given > some > > parameters, e.g. miniD,maxid,id count etc. It learns and adjust from the > > application behavior if not all parameters are given. > > > > So please feel free to incorporate anything you see if or move it to > > contrib. > > > > > > > Further steps towards flexible indexing > > > --------------------------------------- > > > > > > Key: LUCENE-1458 > > > URL: https://issues.apache.org/jira/browse/LUCENE-1458 > > > Project: Lucene - Java > > > Issue Type: New Feature > > > Components: Index > > > Affects Versions: 2.9 > > > Reporter: Michael McCandless > > > Assignee: Michael McCandless > > > Priority: Minor > > > Attachments: LUCENE-1458-back-compat.patch, LUCENE-1458-back- > > compat.patch, LUCENE-1458-back-compat.patch, LUCENE-1458.patch, LUCENE- > > 1458.patch, LUCENE-1458.patch, LUCENE-1458.patch, LUCENE-1458.patch, > > LUCENE-1458.patch, LUCENE-1458.tar.bz2, LUCENE-1458.tar.bz2, LUCENE- > > 1458.tar.bz2, LUCENE-1458.tar.bz2 > > > > > > > > > I attached a very rough checkpoint of my current patch, to get early > > > feedback. All tests pass, though back compat tests don't pass due to > > > changes to package-private APIs plus certain bugs in tests that > > > happened to work (eg call TermPostions.nextPosition() too many times, > > > which the new API asserts against). > > > [Aside: I think, when we commit changes to package-private APIs such > > > that back-compat tests don't pass, we could go back, make a branch on > > > the back-compat tag, commit changes to the tests to use the new > > > package private APIs on that branch, then fix nightly build to use the > > > tip of that branch?o] > > > There's still plenty to do before this is committable! This is a > > > rather large change: > > > * Switches to a new more efficient terms dict format. This still > > > uses tii/tis files, but the tii only stores term & long offset > > > (not a TermInfo). At seek points, tis encodes term & freq/prox > > > offsets absolutely instead of with deltas delta. Also, tis/tii > > > are structured by field, so we don't have to record field number > > > in every term. > > > . > > > On first 1 M docs of Wikipedia, tii file is 36% smaller (0.99 MB > > > -> 0.64 MB) and tis file is 9% smaller (75.5 MB -> 68.5 MB). > > > . > > > RAM usage when loading terms dict index is significantly less > > > since we only load an array of offsets and an array of String (no > > > more TermInfo array). It should be faster to init too. > > > . > > > This part is basically done. > > > * Introduces modular reader codec that strongly decouples terms dict > > > from docs/positions readers. EG there is no more TermInfo used > > > when reading the new format. > > > . > > > There's nice symmetry now between reading & writing in the codec > > > chain -- the current docs/prox format is captured in: > > > {code} > > > FormatPostingsTermsDictWriter/Reader > > > FormatPostingsDocsWriter/Reader (.frq file) and > > > FormatPostingsPositionsWriter/Reader (.prx file). > > > {code} > > > This part is basically done. > > > * Introduces a new "flex" API for iterating through the fields, > > > terms, docs and positions: > > > {code} > > > FieldProducer -> TermsEnum -> DocsEnum -> PostingsEnum > > > {code} > > > This replaces TermEnum/Docs/Positions. SegmentReader emulates the > > > old API on top of the new API to keep back-compat. > > > > > > Next steps: > > > * Plug in new codecs (pulsing, pfor) to exercise the modularity / > > > fix any hidden assumptions. > > > * Expose new API out of IndexReader, deprecate old API but emulate > > > old API on top of new one, switch all core/contrib users to the > > > new API. > > > * Maybe switch to AttributeSources as the base class for TermsEnum, > > > DocsEnum, PostingsEnum -- this would give readers API flexibility > > > (not just index-file-format flexibility). EG if someone wanted > > > to store payload at the term-doc level instead of > > > term-doc-position level, you could just add a new attribute. > > > * Test performance & iterate. > > > > -- > > This message is automatically generated by JIRA. > > - > > You can reply to this email to add a comment to the issue online. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > > For additional commands, e-mail: java-dev-help@lucene.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-dev-help@lucene.apache.org > > --001636ed77c51654bc047570cf4d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi guys:

=A0=A0=A0=A0 What are your thoughts about contributing Kami= kaze as a lucene contrib package? We just finished porting kamikaze to luce= ne 2.9. With the new 2.9 api, it allows us for some more code tuning and op= timization improvements.

=A0=A0=A0=A0 We will be releasing kamikaze, it might a good time to add= it to the lucene contrib package if there is interest.

Thanks
-John

On Thu, Sep 24, 2009 at 6:20 AM, = Uwe Schindler <uwe@= thetaphi.de> wrote:
By the way: In th= e last RC of Lucene 2.9 we added a new method to DocIdSet
called isCacheable(). It is used by e.g. CachingWrapperFilter to determine,=
if a DocIdSet is easy cacheable or must be copied to an OpenBitSetDISI (the=
default is false, so all custom DocIdSets are copied to OpenBitSetDISI by CachingWrapperFilter, even if not needed - if a DocIdSet does not do disk I= O
and have a fast iterator like e.g. the FieldCache ones in
FieldCacheRangeFilter, it should return true; see CHANGES.txt). Maybe this<= br> should also be added to Kamikaze, which is a really nice project! Especiall= y
filter DocIdSets should pass this method to its delegate (see FilterDocIdSe= t
in Lucene).

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de=
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: John Wang (JIRA) [mailto:jira= @apache.org]
> Sent: Thursday, September 24, 2009 3:14 PM
> To: java-dev@lucene.apac= he.org
> Subject: [jira] Commented: (LU= CENE-1458) Further steps towards flexible
> indexing
>
>
> =A0 =A0 [ https://issues.apache.org/jira/browse/LUCENE-
> 1458?page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comment- > tabpanel&focusedCommentId=3D12759112#action_12759112 ]
>
> John Wang commented on LUCENE-1458:
> -----------------------------------
>
> Just a FYI: Kamikaze was originally started as our sandbox for Lucene<= br> > contributions until 2.4 is ready. (we needed the DocIdSet/Iterator
> abstraction that was migrated from Solr)
>
> It has three components:
>
> 1) P4Delta
> 2) Logical boolean operations on DocIdSet/Iterators (I have created a = jira
> ticket and a patch for Lucene awhile ago with performance numbers. It = is
> significantly faster than DisjunctionScorer)
> 3) algorithm to determine which DocIdSet implementations to use given = some
> parameters, e.g. miniD,maxid,id count etc. It learns and adjust from t= he
> application behavior if not all parameters are given.
>
> So please feel free to incorporate anything you see if or move it to > contrib.
>
>
> > Further steps towards flexible indexing
> > ---------------------------------------
> >
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Key: LUCENE-1458
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 URL: https://issues.apache.o= rg/jira/browse/LUCENE-1458
> > =A0 =A0 =A0 =A0 =A0 =A0 Project: Lucene - Java
> > =A0 =A0 =A0 =A0 =A0Issue Type: New Feature
> > =A0 =A0 =A0 =A0 =A0Components: Index
> > =A0 =A0Affects Versions: 2.9
> > =A0 =A0 =A0 =A0 =A0 =A0Reporter: Michael McCandless
> > =A0 =A0 =A0 =A0 =A0 =A0Assignee: Michael McCandless
> > =A0 =A0 =A0 =A0 =A0 =A0Priority: Minor
> > =A0 =A0 =A0 =A0 Attachments: LUCENE-1458-back-compat.patch, LUCEN= E-1458-back-
> compat.patch, LUCENE-1458-back-compat.patch, LUCENE-1458.patch, LUCENE= -
> 1458.patch, LUCENE-1458.patch, LUCENE-1458.patch, LUCENE-1458.patch, > LUCENE-1458.patch, LUCENE-1458.tar.bz2, LUCENE-1458.tar.bz2, LUCENE- > 1458.tar.bz2, LUCENE-1458.tar.bz2
> >
> >
> > I attached a very rough checkpoint of my current patch, to get ea= rly
> > feedback. =A0All tests pass, though back compat tests don't p= ass due to
> > changes to package-private APIs plus certain bugs in tests that > > happened to work (eg call TermPostions.nextPosition() too many ti= mes,
> > which the new API asserts against).
> > [Aside: I think, when we commit changes to package-private APIs s= uch
> > that back-compat tests don't pass, we could go back, make a b= ranch on
> > the back-compat tag, commit changes to the tests to use the new > > package private APIs on that branch, then fix nightly build to us= e the
> > tip of that branch?o]
> > There's still plenty to do before this is committable! This i= s a
> > rather large change:
> > =A0 * Switches to a new more efficient terms dict format. =A0This= still
> > =A0 =A0 uses tii/tis files, but the tii only stores term & lo= ng offset
> > =A0 =A0 (not a TermInfo). =A0At seek points, tis encodes term &am= p; freq/prox
> > =A0 =A0 offsets absolutely instead of with deltas delta. =A0Also,= tis/tii
> > =A0 =A0 are structured by field, so we don't have to record f= ield number
> > =A0 =A0 in every term.
> > .
> > =A0 =A0 On first 1 M docs of Wikipedia, tii file is 36% smaller (= 0.99 MB
> > =A0 =A0 -> 0.64 MB) and tis file is 9% smaller (75.5 MB -> = 68.5 MB).
> > .
> > =A0 =A0 RAM usage when loading terms dict index is significantly = less
> > =A0 =A0 since we only load an array of offsets and an array of St= ring (no
> > =A0 =A0 more TermInfo array). =A0It should be faster to init too.=
> > .
> > =A0 =A0 This part is basically done.
> > =A0 * Introduces modular reader codec that strongly decouples ter= ms dict
> > =A0 =A0 from docs/positions readers. =A0EG there is no more TermI= nfo used
> > =A0 =A0 when reading the new format.
> > .
> > =A0 =A0 There's nice symmetry now between reading & writi= ng in the codec
> > =A0 =A0 chain -- the current docs/prox format is captured in:
> > {code}
> > FormatPostingsTermsDictWriter/Reader
> > FormatPostingsDocsWriter/Reader (.frq file) and
> > FormatPostingsPositionsWriter/Reader (.prx file).
> > {code}
> > =A0 =A0 This part is basically done.
> > =A0 * Introduces a new "flex" API for iterating through= the fields,
> > =A0 =A0 terms, docs and positions:
> > {code}
> > FieldProducer -> TermsEnum -> DocsEnum -> PostingsEnum > > {code}
> > =A0 =A0 This replaces TermEnum/Docs/Positions. =A0SegmentReader e= mulates the
> > =A0 =A0 old API on top of the new API to keep back-compat.
> >
> > Next steps:
> > =A0 * Plug in new codecs (pulsing, pfor) to exercise the modulari= ty /
> > =A0 =A0 fix any hidden assumptions.
> > =A0 * Expose new API out of IndexReader, deprecate old API but em= ulate
> > =A0 =A0 old API on top of new one, switch all core/contrib users = to the
> > =A0 =A0 new API.
> > =A0 * Maybe switch to AttributeSources as the base class for Term= sEnum,
> > =A0 =A0 DocsEnum, PostingsEnum -- this would give readers API fle= xibility
> > =A0 =A0 (not just index-file-format flexibility). =A0EG if someon= e wanted
> > =A0 =A0 to store payload at the term-doc level instead of
> > =A0 =A0 term-doc-position level, you could just add a new attribu= te.
> > =A0 * Test performance & iterate.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------<= br> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


--001636ed77c51654bc047570cf4d--