Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of sean.bridges@gmail.com
 designates 209.85.215.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAL8PwkZCwCNwkwtZbcjrrtEWGVeGVryVRMTbm+osC-ymG_jt_Q@mail.gmail.com>
References: 
 <CAPYoNwdBrzSrCtWcUHdPiGnjo=1QwMcSdj63+RpzzwkuZKAwAA@mail.gmail.com>
 <CAL8PwkZCwCNwkwtZbcjrrtEWGVeGVryVRMTbm+osC-ymG_jt_Q@mail.gmail.com>
From: Sean Bridges <sean.bridges@gmail.com>
Date: Wed, 21 Aug 2013 08:30:26 -0700
Message-ID: 
 <CAOD+0FK1LJN9-rEDfehhaRiUhjVZ-wWZ8CUWoAeedudhYvO9Ag@mail.gmail.com>
Subject: Re: problem found with DiskDocValuesFormat
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=047d7b33daf2180a9204e476dc0f

--047d7b33daf2180a9204e476dc0f
Content-Type: text/plain; charset=ISO-8859-1

What is the recommended way to use DiskDocValuesFormat in production if we
can't reindex when we upgrade?

Will the 4.4 version of DDVF be backwards compatible, or should we make our
own copy of DDVF and give it a different codec name to protect ourselves
against incompatible changes?

Thanks,

Sean


On Tue, Aug 13, 2013 at 4:34 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> DiskDVFormat does not have index back compatibility between minor
> releases; maybe that's what you are seeing?  So, you must fully
> re-index after any DiskDVFormat field after upgrading ...
>
> Only the default formats support index back compatibility between releases.
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Aug 13, 2013 at 4:54 AM, Duke DAI <duke.dai.007@gmail.com> wrote:
> > Hi experts,
> >
> > I'm upgrading Lucene 4.4 and trying to use DocValues instead of store
> field
> > for performance reason. But due to unknown size of index(depends on
> > customer), so I will use DiskDocValuesFormat, especially for some binary
> > field. Then I wrote my customized Codec:
> >
> >       final Codec codec = new Lucene42Codec() {
> >
> >         private final Lucene42DocValuesFormat memoryDVFormat = new
> > Lucene42DocValuesFormat();
> >         private final DiskDocValuesFormat diskDVFormat = new
> > DiskDocValuesFormat();
> >
> >         @Override
> >         public DocValuesFormat getDocValuesFormatForField(String field) {
> >           if
> > (LucenePluginConstants.INDEX_STORED_RETURNABLE_FIELD.equals(field)
> >               || LucenePluginConstants.PAYLOAD_FIELD_NAME.equals(field)
> ||
> > LucenePluginConstants.INDEX_NODE_ID_DOC_VALUE.equals(field)) {
> >             return diskDVFormat;
> >           } else {
> >             return memoryDVFormat
> >           }
> >         }
> >       };
> >       iwc.setCodec(codec);
> >
> > Here field LucenePluginConstants.INDEX_NODE_ID_DOC_VALUE is numeric
> field,
> > long type. And others are binary.
> >
> > Then I consume DV like below pseudo-code:
> >     nodeIDDocValuesSource =
> >             MultiDocValues.getNumericValues(searcher.getIndexReader(),
> >                 LucenePluginConstants.INDEX_NODE_ID_DOC_VALUE);
> >
> >    ......
> >    long nodeId= nodeIDDocValuesSource.get(scoreDoc.doc);
> >
> > Then I'm sure I get a wrong nodeId, which will be verified by upper logic
> > and treated as data corruption.
> >
> >
> > But if I change to memoryDVFormat for the long type field, then
> everything
> > is OK.
> >
> > Also for upgrading legacy data, I keep two index format, DV or stored
> > field, controlled by version. If I use stored field, everything is OK.
> > So I guess there is a bug with  DiskDocValuesFormat, numeric data type,
> > does it relate to byte-aligned numeric compression?
> > Or I didn't use DiskDocValuesFormat correctly? Seems no other parameters
> > for it.
> >
> > Sorry that I have no pure Lucene test case yet. Hope someone shed some
> > light on this.
> >
> >
> >
> >
> > Best regards,
> > Duke
> > If not now, when? If not me, who?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--047d7b33daf2180a9204e476dc0f--