Thanks for the heads-up, TCK. The Dietz & Sleator article I found at 
http://www.cs.cmu.edu/~sleator/papers/maintaining-order.pdf
looks very interesting.

String sorting in Lucene is indeed fairly expensive and we've experimented with
two solutions to this, none of which are golden bullets.

1) Store the order, not the content.

Like the Lucene default sorter, we fetch all the terms for the sort field on first
sort. The docIDs are mapped to the terms and an complete sort is performed 
(by using a custom Collator: We want to avoid using CollationKeys as they
take too much memory. But that's another story). When the sort is finished,
we have an integer-array where each entry is the relative position for a given
document (referenced by docID). We then de-reference the loaded terms
so that the memory is freed.

Determining the order of two documents after the initial build is extremely
simple: order[docIDa] - order[docIDb] (or maybe it was the other way 
around, I forget).

Cons: Large temporarily memory overhead upon initial sort, though not as 
much as using CollationKeys. Slower initial sort than Lucene build-in.
Pros: Very fast subsequent sorts, very low memory-overhead when running.
Joker: This approach could be used to perform a post-processing on a 
generated index and storing the orders as a file along with the index, then 
pushing the complete package to searchers. This would lower the memory
requirements for the searchers significantly.


2) As above, but use a multipass order builder.

For the first sort, the order must be calculated: A sliding window sorter is created 
with a buffer of a fixed size. The window slides through all terms for the sort field 
and extracts the top X. The order-array is updated for the documents that has 
one of these terms. The sliding is repeated multiple times, where terms ordered 
before the last term of the previous iteration are ignored.

Cons: _Very_ slow (too slow in the current implementation) order build.
Pros: Same as above.
Joker: The buffer size determines memory use vs. order build time.


The multipass approach looks promising, but requires more work to get to a
usable state. Right now it takes minutes to build the order-array for half a 
million documents, with a buffer size requiring 5 iterations. If I ever get it to
work, I'll be sure to share it.

Regards,
Toke Eskildsen

________________________________________
From: TCK [moonwatcher32329@gmail.com]
Sent: 09 December 2009 22:58
To: java-user@lucene.apache.org
Subject: Re: heap memory issues when sorting by a string field

Thanks Mike for opening this jira ticket and for your patch. Explicitly
removing the entry from the WHM definitely does reduce the number of GC
cycles taken to free the huge StringIndex objects that get created when
doing a sort by a string field.

But I'm still trying to figure out why it is necessary for lucene to load
into memory the sorted-by field of every single document in the index (which
is what makes the StringIndex objects so big) on the first query. Why isn't
it sufficient to load the field values for only the documents that match the
given search criteria? Perhaps by using an order-maintenance data structure
(Dietz&Sleator, or Bender et al) coupled with a balanced search tree,
instead of the simple lookup and order arrays currently used in StringIndex,
we can dynamically grow it as needed by successive queries rather than
loading everything on the first query.

Apologies if this is a naive question... I'm a newbie to the lucene code and
can't wait for the new lucene book to come out:-)

-TCK




On Tue, Dec 8, 2009 at 5:43 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> I've opened LUCENE-2135.
>
> Mike
>
> On Tue, Dec 8, 2009 at 5:36 AM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
> > This is a rather disturbing implementation detail of WeakHashMap, that
> > it needs the one extra step (invoking one of its methods) for its weak
> > keys to be reclaimable.
> >
> > Maybe on IndexReader.close(), Lucene should go and evict all entries
> > in the FieldCache associated with that reader.  Ie, step through the
> > sub-readers, and if they are truly closed as well (not shared w/ other
> > readers), evict.  I'll open an issue.
> >
> > Even in TCK's code fragment, it's not until the final line is done
> > executing, that the cache key even loses all hard references, because
> > it's that line that assigns to sSearcher, replacing the strong
> > reference to the old searcher.  Inserting sSearcher = null prior to
> > that would drop the hard reference sooner, but because of this impl
> > detail of WeakHashMap, something would still have to touch it (eg, a
> > warmup query that hits the field cache) before it's reclaimable.
> >
> > Mike
> >
> > On Mon, Dec 7, 2009 at 7:38 PM, Tom Hill <solr-list@worldware.com>
> wrote:
> >> Hi -
> >>
> >> If I understand correctly, WeakHashMap does not free the memory for the
> >> value (cached data) when the key is nulled, or even when the key is
> garbage
> >> collected.
> >>
> >> It requires one more step: a method on WeakHashMap must be called to
> allow
> >> it to release its hard reference to the cached data. It appears that
> most
> >> methods in WeakHashMap end up calling expungeStaleEntries, which will
> clear
> >> the hard reference. But you have to call some method on the map, before
> the
> >> memory is eligible for garbage collection.
> >>
> >> So it requires four stages to free the cached data. Null the key; A GC
> to
> >> release the weak reference to the key; A call to some method on the map;
> >> Then the next GC cycle should free the value.
> >>
> >> So it seems possible that you could end up with double memory usage for
> a
> >> time. If you don't have a GC between the time that you close the old
> reader,
> >> and you start to load the field cache entry for the next reader, then
> the
> >> key may still be hanging around uncollected.
> >>
> >> At that point, it may run a GC when you allocate the new cache, but
> that's
> >> only the first GC. It can't free the cached data until after the next
> call
> >> to expungeStaleEntries, so for a while you have both caches around.
> >>
> >> This extra usage could cause things to move into tenured space. Could
> this
> >> be causing your problem?
> >>
> >> Workaround would be to cause some method to be called on the
> WeakHashMap.
> >> You don't want to call get(), since that will try to populate the cache.
> >> Maybe if you tried putting a small value to the cache, and doing a GC,
> and
> >> see if your memory drops then.
> >>
> >>
> >> Tom
> >>
> >>
> >>
> >> On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@gmail.com> wrote:
> >>
> >>> Thanks for the response. But I'm definitely calling close() on the old
> >>> reader and opening a new one (not using reopen). Also, to simplify the
> >>> analysis, I did my test with a single-threaded requester to eliminate
> any
> >>> concurrency issues.
> >>>
> >>> I'm doing:
> >>> sSearcher.getIndexReader().close();
> >>> sSearcher.close(); // this actually seems to be a no-op
> >>> IndexReader newIndexReader = IndexReader.open(newDirectory);
> >>> sSearcher = new IndexSearcher(newIndexReader);
> >>>
> >>> Btw, isn't it bad practice anyway to have an unbounded cache? Are there
> any
> >>> plans to replace the HashMaps used for the innerCaches with an actual
> >>> size-bounded cache with some eviction policy (perhaps EhCache or
> something)
> >>> ?
> >>>
> >>> Thanks again,
> >>> TCK
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <
> erickerickson@gmail.com
> >>> >wrote:
> >>>
> >>> > What this sounds like is that you're not really closing your
> >>> > readers even though you think you are. Sorting indeed uses up
> >>> > significant memory when it populates internal caches and keeps
> >>> > it around for later use (which is one of the reasons that warming
> >>> > queries matter). But if you really do close the reader, I'm pretty
> >>> > sure the memory should be GC-able.
> >>> >
> >>> > One thing that trips people up is IndexReader.reopen(). If it
> >>> > returns a reader different than the original, you *must* close the
> >>> > old one. If you don't, the old reader is still hanging around and
> >>> > memory won't be returne.... An example from the Javadocs...
> >>> >
> >>> >  IndexReader reader = ...
> >>> >  ...
> >>> >  IndexReader new = r.reopen();
> >>> >  if (new != reader) {
> >>> >   ...     // reader was reopened
> >>> >   reader.close();
> >>> >  }
> >>> >  reader = new;
> >>> >  ...
> >>> >
> >>> >
> >>> > If this is irrelevant, could you post your close/open
> >>> >
> >>> > code?
> >>> >
> >>> > HTH
> >>> >
> >>> > Erick
> >>> >
> >>> >
> >>> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@gmail.com>
> wrote:
> >>> >
> >>> > > Hi,
> >>> > > I'm having heap memory issues when I do lucene queries involving
> >>> sorting
> >>> > by
> >>> > > a string field. Such queries seem to load a lot of data in to
the
> heap.
> >>> > > Moreover lucene seems to hold on to references to this data even
> after
> >>> > the
> >>> > > index reader has been closed and a full GC has been run. Some
of
> the
> >>> > > consequences of this are that in my generational heap configuration
> a
> >>> lot
> >>> > > of
> >>> > > memory gets promoted to tenured space each time I close the old
> index
> >>> > > reader
> >>> > > and after opening and querying using a new one, and the tenured
> space
> >>> > > eventually gets fragmented causing a lot of promotion failures
> >>> resulting
> >>> > in
> >>> > > jvm hangs while the jvm does stop-the-world GCs.
> >>> > >
> >>> > > Does anyone know any workarounds to avoid these memory issues
when
> >>> doing
> >>> > > such lucene queries?
> >>> > >
> >>> > > My profiling showed that even after a full GC lucene is holding
on
> to a
> >>> > lot
> >>> > > of references to field value data notably via the
> >>> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the
> WeakHashMap
> >>> > > readerCaches are using unbounded HashMaps as the innerCaches and
I
> used
> >>> > > reflection to replace these innerCaches with dummy empty HashMaps,
> but
> >>> > > still
> >>> > > I'm seeing the same behavior. I wondered if anyone has gone through
> >>> these
> >>> > > same issues before and would offer any advice.
> >>> > >
> >>> > > Thanks a lot,
> >>> > > TCK
> >>> > >
> >>> >
> >>>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

It's not that it's "necessary" -- this is just how Lucene's sorting
has always worked ;)  But, it's just software!  You could whip up a
patch...

I'm not familiar with the order-maintenance problem & solutions
offhand, but it certainly sounds interesting.

One issue is that loading only certain values on demand (from disk)
can be costly, eg it requires multiple disk head seeks (though, SSDs
will help here, though it's still maybe several order of magnitudes
slower than RAM (as long as the RAM isn't swapped out).

I think Lucene could cut back on the RAM usage by more compactly
representing the values.  Making separate String per value is alot of
overhead...

Mike

On Wed, Dec 9, 2009 at 4:58 PM, TCK <moonwatcher32329@gmail.com> wrote:
> Thanks Mike for opening this jira ticket and for your patch. Explicitly
> removing the entry from the WHM definitely does reduce the number of GC
> cycles taken to free the huge StringIndex objects that get created when
> doing a sort by a string field.
>
> But I'm still trying to figure out why it is necessary for lucene to load
> into memory the sorted-by field of every single document in the index (which
> is what makes the StringIndex objects so big) on the first query. Why isn't
> it sufficient to load the field values for only the documents that match the
> given search criteria? Perhaps by using an order-maintenance data structure
> (Dietz&Sleator, or Bender et al) coupled with a balanced search tree,
> instead of the simple lookup and order arrays currently used in StringIndex,
> we can dynamically grow it as needed by successive queries rather than
> loading everything on the first query.
>
> Apologies if this is a naive question... I'm a newbie to the lucene code and
> can't wait for the new lucene book to come out:-)
>
> -TCK
>
>
>
>
> On Tue, Dec 8, 2009 at 5:43 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> I've opened LUCENE-2135.
>>
>> Mike
>>
>> On Tue, Dec 8, 2009 at 5:36 AM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>> > This is a rather disturbing implementation detail of WeakHashMap, that
>> > it needs the one extra step (invoking one of its methods) for its weak
>> > keys to be reclaimable.
>> >
>> > Maybe on IndexReader.close(), Lucene should go and evict all entries
>> > in the FieldCache associated with that reader.  Ie, step through the
>> > sub-readers, and if they are truly closed as well (not shared w/ other
>> > readers), evict.  I'll open an issue.
>> >
>> > Even in TCK's code fragment, it's not until the final line is done
>> > executing, that the cache key even loses all hard references, because
>> > it's that line that assigns to sSearcher, replacing the strong
>> > reference to the old searcher.  Inserting sSearcher = null prior to
>> > that would drop the hard reference sooner, but because of this impl
>> > detail of WeakHashMap, something would still have to touch it (eg, a
>> > warmup query that hits the field cache) before it's reclaimable.
>> >
>> > Mike
>> >
>> > On Mon, Dec 7, 2009 at 7:38 PM, Tom Hill <solr-list@worldware.com>
>> wrote:
>> >> Hi -
>> >>
>> >> If I understand correctly, WeakHashMap does not free the memory for the
>> >> value (cached data) when the key is nulled, or even when the key is
>> garbage
>> >> collected.
>> >>
>> >> It requires one more step: a method on WeakHashMap must be called to
>> allow
>> >> it to release its hard reference to the cached data. It appears that
>> most
>> >> methods in WeakHashMap end up calling expungeStaleEntries, which will
>> clear
>> >> the hard reference. But you have to call some method on the map, before
>> the
>> >> memory is eligible for garbage collection.
>> >>
>> >> So it requires four stages to free the cached data. Null the key; A GC
>> to
>> >> release the weak reference to the key; A call to some method on the map;
>> >> Then the next GC cycle should free the value.
>> >>
>> >> So it seems possible that you could end up with double memory usage for
>> a
>> >> time. If you don't have a GC between the time that you close the old
>> reader,
>> >> and you start to load the field cache entry for the next reader, then
>> the
>> >> key may still be hanging around uncollected.
>> >>
>> >> At that point, it may run a GC when you allocate the new cache, but
>> that's
>> >> only the first GC. It can't free the cached data until after the next
>> call
>> >> to expungeStaleEntries, so for a while you have both caches around.
>> >>
>> >> This extra usage could cause things to move into tenured space. Could
>> this
>> >> be causing your problem?
>> >>
>> >> Workaround would be to cause some method to be called on the
>> WeakHashMap.
>> >> You don't want to call get(), since that will try to populate the cache.
>> >> Maybe if you tried putting a small value to the cache, and doing a GC,
>> and
>> >> see if your memory drops then.
>> >>
>> >>
>> >> Tom
>> >>
>> >>
>> >>
>> >> On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@gmail.com> wrote:
>> >>
>> >>> Thanks for the response. But I'm definitely calling close() on the old
>> >>> reader and opening a new one (not using reopen). Also, to simplify the
>> >>> analysis, I did my test with a single-threaded requester to eliminate
>> any
>> >>> concurrency issues.
>> >>>
>> >>> I'm doing:
>> >>> sSearcher.getIndexReader().close();
>> >>> sSearcher.close(); // this actually seems to be a no-op
>> >>> IndexReader newIndexReader = IndexReader.open(newDirectory);
>> >>> sSearcher = new IndexSearcher(newIndexReader);
>> >>>
>> >>> Btw, isn't it bad practice anyway to have an unbounded cache? Are there
>> any
>> >>> plans to replace the HashMaps used for the innerCaches with an actual
>> >>> size-bounded cache with some eviction policy (perhaps EhCache or
>> something)
>> >>> ?
>> >>>
>> >>> Thanks again,
>> >>> TCK
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <
>> erickerickson@gmail.com
>> >>> >wrote:
>> >>>
>> >>> > What this sounds like is that you're not really closing your
>> >>> > readers even though you think you are. Sorting indeed uses up
>> >>> > significant memory when it populates internal caches and keeps
>> >>> > it around for later use (which is one of the reasons that warming
>> >>> > queries matter). But if you really do close the reader, I'm pretty
>> >>> > sure the memory should be GC-able.
>> >>> >
>> >>> > One thing that trips people up is IndexReader.reopen(). If it
>> >>> > returns a reader different than the original, you *must* close
the
>> >>> > old one. If you don't, the old reader is still hanging around and
>> >>> > memory won't be returne.... An example from the Javadocs...
>> >>> >
>> >>> >  IndexReader reader = ...
>> >>> >  ...
>> >>> >  IndexReader new = r.reopen();
>> >>> >  if (new != reader) {
>> >>> >   ...     // reader was reopened
>> >>> >   reader.close();
>> >>> >  }
>> >>> >  reader = new;
>> >>> >  ...
>> >>> >
>> >>> >
>> >>> > If this is irrelevant, could you post your close/open
>> >>> >
>> >>> > code?
>> >>> >
>> >>> > HTH
>> >>> >
>> >>> > Erick
>> >>> >
>> >>> >
>> >>> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@gmail.com>
>> wrote:
>> >>> >
>> >>> > > Hi,
>> >>> > > I'm having heap memory issues when I do lucene queries involving
>> >>> sorting
>> >>> > by
>> >>> > > a string field. Such queries seem to load a lot of data in
to the
>> heap.
>> >>> > > Moreover lucene seems to hold on to references to this data
even
>> after
>> >>> > the
>> >>> > > index reader has been closed and a full GC has been run. Some
of
>> the
>> >>> > > consequences of this are that in my generational heap configuration
>> a
>> >>> lot
>> >>> > > of
>> >>> > > memory gets promoted to tenured space each time I close the
old
>> index
>> >>> > > reader
>> >>> > > and after opening and querying using a new one, and the tenured
>> space
>> >>> > > eventually gets fragmented causing a lot of promotion failures
>> >>> resulting
>> >>> > in
>> >>> > > jvm hangs while the jvm does stop-the-world GCs.
>> >>> > >
>> >>> > > Does anyone know any workarounds to avoid these memory issues
when
>> >>> doing
>> >>> > > such lucene queries?
>> >>> > >
>> >>> > > My profiling showed that even after a full GC lucene is holding
on
>> to a
>> >>> > lot
>> >>> > > of references to field value data notably via the
>> >>> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the
>> WeakHashMap
>> >>> > > readerCaches are using unbounded HashMaps as the innerCaches
and I
>> used
>> >>> > > reflection to replace these innerCaches with dummy empty HashMaps,
>> but
>> >>> > > still
>> >>> > > I'm seeing the same behavior. I wondered if anyone has gone
through
>> >>> these
>> >>> > > same issues before and would offer any advice.
>> >>> > >
>> >>> > > Thanks a lot,
>> >>> > > TCK
>> >>> > >
>> >>> >
>> >>>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Yes, you found it!  Is that what you're hitting?

I don't know of a workaround though... this is just how SpanQuery
currently works...

Mike

On Wed, Dec 9, 2009 at 4:56 PM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
> Mike,
>
> Is this the thread?
>
> http://www.lucidimagination.com/search/document/1e87d488a904b89f/spannearquery_s_spans_payloads#8103efdc9705a763
>
> Maybe we need a recommended workaround for this?
>
> Jason
>
> On Wed, Dec 9, 2009 at 1:17 PM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>> That sounds familiar... try to track down the last thread maybe?
>>
>> I think it was this: if the payload was already retrieved for a prior
>> span then the current span won't be able to retrieve it, so even
>> though you know a payload falls within the span you're looking at, you
>> won't get it back, if it already fell on a prior span.
>>
>> Mike
>>
>> On Wed, Dec 9, 2009 at 11:25 AM, Jason Rutherglen
>> <jason.rutherglen@gmail.com> wrote:
>>> Right we're getting the spans, however it's just the payloads that are
>>> missing, randomly...
>>>
>>> On Wed, Dec 9, 2009 at 2:23 AM, Michael McCandless
>>> <lucene@mikemccandless.com> wrote:
>>>> There was a thread a while back about how span queries don't enumerate
>>>> every possible span, but I can't remember if that included sometimes
>>>> missing payloads...
>>>>
>>>> Mike
>>>>
>>>> On Tue, Dec 8, 2009 at 7:34 PM, Jason Rutherglen
>>>> <jason.rutherglen@gmail.com> wrote:
>>>>> Howdy,
>>>>>
>>>>> I am wondering if anyone has seen
>>>>> NearSpansUnordered.getPayload() not return payloads that are
>>>>> verifiably accessible via IR.termPositions? It's a bit confusing
>>>>> because most of the time they're returned properly.
>>>>>
>>>>> I suspect the payload logic gets tripped up in
>>>>> NearSpansUnordered. I'll put together a test case, however the
>>>>> difficulty is that we're only seeing the issue with largish 800
>>>>> MB indexes, which could make the test case a little crazy.
>>>>>
>>>>> Jason
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

This is a bug in InstantiatedIndex. The termDoc(null) was added to get all
documents. This was never implemented in Instantiated Index. Can you open an
issue?

There maybe other queries fail because of this (e.g.
FieldCacheRangeFilter,...).

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Jason Fennell [mailto:jdfennell@gmail.com]
> Sent: Wednesday, December 09, 2009 7:48 PM
> To: java-user@lucene.apache.org
> Subject: MatchAllDocsQuery and InstantiatedIndex on Lucene 2.9.1
> 
> I'm trying to upgrade our application from Lucene 2.4.1 to Lucene 2.9.1.
> I've been using an InstantiatedIndex to do a bunch of unit testing, but am
> running into a some problems with Lucene 2.9.1.
> In particular, when I try to run a MatchAllDocsQuery on my
> InstantiatedIndex
> (which worked fine on 2.4.1) a NullPointerException is raised.  I think I
> tracked down the problem in the Lucene source.
> 
> In the 2.9.1 MatchAllDocsQuery, the MatchAllScorer retrieves termDocs from
> the IndexReader passed to it with
> 
> reader.termDocs(null)
> 
> which I assume is supposed to return all termDocs.  However, tracing this
> call down
> 
> IndexReader.termDocs(term) calls InstantiatedTermDocs.seek(
> term) calls InstantiatedIndex.findTerm(term)
> 
> which is implemented as
> 
> InstantiatedTerm findTerm(Term term) {
>     return findTerm(term.field(), term.text());
> }
> 
> which, since term is null, results in a NullPointerException.  This seems
> to
> me like an bug either in the MatchAllDocsQuery implementation (the version
> in 2.4.1 did not use termDocs, so did not pass through this null), or a
> bug
> in the implementation of InstantiatedIndex.
> 
> Any suggestions on what I can do about this?  I definitely can't get rid
> of
> the MatchAllDocsQuery and don't really want to move back to a slow
> RAMDirectory.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Thanks Mike for opening this jira ticket and for your patch. Explicitly
removing the entry from the WHM definitely does reduce the number of GC
cycles taken to free the huge StringIndex objects that get created when
doing a sort by a string field.

But I'm still trying to figure out why it is necessary for lucene to load
into memory the sorted-by field of every single document in the index (which
is what makes the StringIndex objects so big) on the first query. Why isn't
it sufficient to load the field values for only the documents that match the
given search criteria? Perhaps by using an order-maintenance data structure
(Dietz&Sleator, or Bender et al) coupled with a balanced search tree,
instead of the simple lookup and order arrays currently used in StringIndex,
we can dynamically grow it as needed by successive queries rather than
loading everything on the first query.

Apologies if this is a naive question... I'm a newbie to the lucene code and
can't wait for the new lucene book to come out:-)

-TCK




On Tue, Dec 8, 2009 at 5:43 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> I've opened LUCENE-2135.
>
> Mike
>
> On Tue, Dec 8, 2009 at 5:36 AM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
> > This is a rather disturbing implementation detail of WeakHashMap, that
> > it needs the one extra step (invoking one of its methods) for its weak
> > keys to be reclaimable.
> >
> > Maybe on IndexReader.close(), Lucene should go and evict all entries
> > in the FieldCache associated with that reader.  Ie, step through the
> > sub-readers, and if they are truly closed as well (not shared w/ other
> > readers), evict.  I'll open an issue.
> >
> > Even in TCK's code fragment, it's not until the final line is done
> > executing, that the cache key even loses all hard references, because
> > it's that line that assigns to sSearcher, replacing the strong
> > reference to the old searcher.  Inserting sSearcher = null prior to
> > that would drop the hard reference sooner, but because of this impl
> > detail of WeakHashMap, something would still have to touch it (eg, a
> > warmup query that hits the field cache) before it's reclaimable.
> >
> > Mike
> >
> > On Mon, Dec 7, 2009 at 7:38 PM, Tom Hill <solr-list@worldware.com>
> wrote:
> >> Hi -
> >>
> >> If I understand correctly, WeakHashMap does not free the memory for the
> >> value (cached data) when the key is nulled, or even when the key is
> garbage
> >> collected.
> >>
> >> It requires one more step: a method on WeakHashMap must be called to
> allow
> >> it to release its hard reference to the cached data. It appears that
> most
> >> methods in WeakHashMap end up calling expungeStaleEntries, which will
> clear
> >> the hard reference. But you have to call some method on the map, before
> the
> >> memory is eligible for garbage collection.
> >>
> >> So it requires four stages to free the cached data. Null the key; A GC
> to
> >> release the weak reference to the key; A call to some method on the map;
> >> Then the next GC cycle should free the value.
> >>
> >> So it seems possible that you could end up with double memory usage for
> a
> >> time. If you don't have a GC between the time that you close the old
> reader,
> >> and you start to load the field cache entry for the next reader, then
> the
> >> key may still be hanging around uncollected.
> >>
> >> At that point, it may run a GC when you allocate the new cache, but
> that's
> >> only the first GC. It can't free the cached data until after the next
> call
> >> to expungeStaleEntries, so for a while you have both caches around.
> >>
> >> This extra usage could cause things to move into tenured space. Could
> this
> >> be causing your problem?
> >>
> >> Workaround would be to cause some method to be called on the
> WeakHashMap.
> >> You don't want to call get(), since that will try to populate the cache.
> >> Maybe if you tried putting a small value to the cache, and doing a GC,
> and
> >> see if your memory drops then.
> >>
> >>
> >> Tom
> >>
> >>
> >>
> >> On Mon, Dec 7, 2009 at 1:48 PM, TCK <moonwatcher32329@gmail.com> wrote:
> >>
> >>> Thanks for the response. But I'm definitely calling close() on the old
> >>> reader and opening a new one (not using reopen). Also, to simplify the
> >>> analysis, I did my test with a single-threaded requester to eliminate
> any
> >>> concurrency issues.
> >>>
> >>> I'm doing:
> >>> sSearcher.getIndexReader().close();
> >>> sSearcher.close(); // this actually seems to be a no-op
> >>> IndexReader newIndexReader = IndexReader.open(newDirectory);
> >>> sSearcher = new IndexSearcher(newIndexReader);
> >>>
> >>> Btw, isn't it bad practice anyway to have an unbounded cache? Are there
> any
> >>> plans to replace the HashMaps used for the innerCaches with an actual
> >>> size-bounded cache with some eviction policy (perhaps EhCache or
> something)
> >>> ?
> >>>
> >>> Thanks again,
> >>> TCK
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Dec 7, 2009 at 4:37 PM, Erick Erickson <
> erickerickson@gmail.com
> >>> >wrote:
> >>>
> >>> > What this sounds like is that you're not really closing your
> >>> > readers even though you think you are. Sorting indeed uses up
> >>> > significant memory when it populates internal caches and keeps
> >>> > it around for later use (which is one of the reasons that warming
> >>> > queries matter). But if you really do close the reader, I'm pretty
> >>> > sure the memory should be GC-able.
> >>> >
> >>> > One thing that trips people up is IndexReader.reopen(). If it
> >>> > returns a reader different than the original, you *must* close the
> >>> > old one. If you don't, the old reader is still hanging around and
> >>> > memory won't be returne.... An example from the Javadocs...
> >>> >
> >>> >  IndexReader reader = ...
> >>> >  ...
> >>> >  IndexReader new = r.reopen();
> >>> >  if (new != reader) {
> >>> >   ...     // reader was reopened
> >>> >   reader.close();
> >>> >  }
> >>> >  reader = new;
> >>> >  ...
> >>> >
> >>> >
> >>> > If this is irrelevant, could you post your close/open
> >>> >
> >>> > code?
> >>> >
> >>> > HTH
> >>> >
> >>> > Erick
> >>> >
> >>> >
> >>> > On Mon, Dec 7, 2009 at 4:27 PM, TCK <moonwatcher32329@gmail.com>
> wrote:
> >>> >
> >>> > > Hi,
> >>> > > I'm having heap memory issues when I do lucene queries involving
> >>> sorting
> >>> > by
> >>> > > a string field. Such queries seem to load a lot of data in to
the
> heap.
> >>> > > Moreover lucene seems to hold on to references to this data even
> after
> >>> > the
> >>> > > index reader has been closed and a full GC has been run. Some
of
> the
> >>> > > consequences of this are that in my generational heap configuration
> a
> >>> lot
> >>> > > of
> >>> > > memory gets promoted to tenured space each time I close the old
> index
> >>> > > reader
> >>> > > and after opening and querying using a new one, and the tenured
> space
> >>> > > eventually gets fragmented causing a lot of promotion failures
> >>> resulting
> >>> > in
> >>> > > jvm hangs while the jvm does stop-the-world GCs.
> >>> > >
> >>> > > Does anyone know any workarounds to avoid these memory issues
when
> >>> doing
> >>> > > such lucene queries?
> >>> > >
> >>> > > My profiling showed that even after a full GC lucene is holding
on
> to a
> >>> > lot
> >>> > > of references to field value data notably via the
> >>> > > FieldCacheImpl/ExtendedFieldCacheImpl. I noticed that the
> WeakHashMap
> >>> > > readerCaches are using unbounded HashMaps as the innerCaches and
I
> used
> >>> > > reflection to replace these innerCaches with dummy empty HashMaps,
> but
> >>> > > still
> >>> > > I'm seeing the same behavior. I wondered if anyone has gone through
> >>> these
> >>> > > same issues before and would offer any advice.
> >>> > >
> >>> > > Thanks a lot,
> >>> > > TCK
> >>> > >
> >>> >
> >>>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mike,

Is this the thread?

http://www.lucidimagination.com/search/document/1e87d488a904b89f/spannearquery_s_spans_payloads#8103efdc9705a763

Maybe we need a recommended workaround for this?

Jason

On Wed, Dec 9, 2009 at 1:17 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> That sounds familiar... try to track down the last thread maybe?
>
> I think it was this: if the payload was already retrieved for a prior
> span then the current span won't be able to retrieve it, so even
> though you know a payload falls within the span you're looking at, you
> won't get it back, if it already fell on a prior span.
>
> Mike
>
> On Wed, Dec 9, 2009 at 11:25 AM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
>> Right we're getting the spans, however it's just the payloads that are
>> missing, randomly...
>>
>> On Wed, Dec 9, 2009 at 2:23 AM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>> There was a thread a while back about how span queries don't enumerate
>>> every possible span, but I can't remember if that included sometimes
>>> missing payloads...
>>>
>>> Mike
>>>
>>> On Tue, Dec 8, 2009 at 7:34 PM, Jason Rutherglen
>>> <jason.rutherglen@gmail.com> wrote:
>>>> Howdy,
>>>>
>>>> I am wondering if anyone has seen
>>>> NearSpansUnordered.getPayload() not return payloads that are
>>>> verifiably accessible via IR.termPositions? It's a bit confusing
>>>> because most of the time they're returned properly.
>>>>
>>>> I suspect the payload logic gets tripped up in
>>>> NearSpansUnordered. I'll put together a test case, however the
>>>> difficulty is that we're only seeing the issue with largish 800
>>>> MB indexes, which could make the test case a little crazy.
>>>>
>>>> Jason
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

That sounds familiar... try to track down the last thread maybe?

I think it was this: if the payload was already retrieved for a prior
span then the current span won't be able to retrieve it, so even
though you know a payload falls within the span you're looking at, you
won't get it back, if it already fell on a prior span.

Mike

On Wed, Dec 9, 2009 at 11:25 AM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
> Right we're getting the spans, however it's just the payloads that are
> missing, randomly...
>
> On Wed, Dec 9, 2009 at 2:23 AM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>> There was a thread a while back about how span queries don't enumerate
>> every possible span, but I can't remember if that included sometimes
>> missing payloads...
>>
>> Mike
>>
>> On Tue, Dec 8, 2009 at 7:34 PM, Jason Rutherglen
>> <jason.rutherglen@gmail.com> wrote:
>>> Howdy,
>>>
>>> I am wondering if anyone has seen
>>> NearSpansUnordered.getPayload() not return payloads that are
>>> verifiably accessible via IR.termPositions? It's a bit confusing
>>> because most of the time they're returned properly.
>>>
>>> I suspect the payload logic gets tripped up in
>>> NearSpansUnordered. I'll put together a test case, however the
>>> difficulty is that we're only seeing the issue with largish 800
>>> MB indexes, which could make the test case a little crazy.
>>>
>>> Jason
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hi,
I have a HitCollector that processes all hits from a query.  I want all
hits, not the top N hits.  I am converting my HitCollector to a Collector
for Lucene 3.0.0, and I'm a little confused by the new interface.

I assume that I can implement by new Collector much like the code on the API
Docs:

 Searcher searcher = new IndexSearcher(indexReader);
 final BitSet bits = new BitSet(indexReader.maxDoc());
 searcher.search(query, new Collector() {
   private int docBase;

   *// ignore scorer*
   public void setScorer(Scorer scorer) {
   }

   *// accept docs out of order (for a BitSet it doesn't matter)*
   public boolean acceptsDocsOutOfOrder() {
     return true;
   }

   public void collect(int doc) {
     bits.set(doc + docBase);
   }

   public void setNextReader(IndexReader reader, int docBase) {
     this.docBase = docBase;
   }
 });


But I'm confused what the docBasing is (I need to get fields from each
document from my index searcher).  Do I need to use the doc base or
setNextReader?  Also, what is the purpose of acceptsDocsOutOfOrder?  I
see the docs note on it but I'm not sure how I could apply that or if
I should care about it.

Thanks,
Max

I'm trying to upgrade our application from Lucene 2.4.1 to Lucene 2.9.1.
I've been using an InstantiatedIndex to do a bunch of unit testing, but am
running into a some problems with Lucene 2.9.1.
In particular, when I try to run a MatchAllDocsQuery on my InstantiatedIndex
(which worked fine on 2.4.1) a NullPointerException is raised.  I think I
tracked down the problem in the Lucene source.

In the 2.9.1 MatchAllDocsQuery, the MatchAllScorer retrieves termDocs from
the IndexReader passed to it with

reader.termDocs(null)

which I assume is supposed to return all termDocs.  However, tracing this
call down

IndexReader.termDocs(term) calls InstantiatedTermDocs.seek(
term) calls InstantiatedIndex.findTerm(term)

which is implemented as

InstantiatedTerm findTerm(Term term) {
    return findTerm(term.field(), term.text());
}

which, since term is null, results in a NullPointerException.  This seems to
me like an bug either in the MatchAllDocsQuery implementation (the version
in 2.4.1 did not use termDocs, so did not pass through this null), or a bug
in the implementation of InstantiatedIndex.

Any suggestions on what I can do about this?  I definitely can't get rid of
the MatchAllDocsQuery and don't really want to move back to a slow
RAMDirectory.

On Fri, Oct 02, 2009 at 11:40:09PM -0700, m.harig wrote:
> 
> Thanks Uwe Schindler ,
> 
>       If i use an IndexReader[] to use MultiReader  , will it be thread
> safe? because i've to reopen my IndexReader to check whether my index is
> updated or not . In this case how do i handle it? please suggest me

Hi,

AFAIK readers are cloned when they are used inside MultiReader, so I
guess reopening a single reader outside MultiReader has no effect and
reopen on MultiReader reopen all the readers.

So I think there is no way to reopen one specific reader inside a
MultiReader without a new instance of the MultiReader.

IMHO the easy way is to use MultiReader as a normal reader (with
default constructor MultiReader(IndexReader... readers) ), forget the
original readers and use reopen on the MultiReader. It should be
thread-safe.
You should check the sources for more complicated usage.

> 
> 
> 
> Uwe Schindler wrote:
> > 
> > Use MultiReader which is an IndexReader on top of various
> > Sub-IndexReaders.
> > 
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> > 
> >> -----Original Message-----
> >> From: m.harig [mailto:m.harig@gmail.com]
> >> Sent: Friday, October 02, 2009 6:52 PM
> >> To: java-user@lucene.apache.org
> >> Subject: index reader for multiple indexes
> >> 
> >> 
> >> hello all ,
> >> 
> >>        am merging more than one indexes to search a document , how do i
> >> use
> >> IndexReader here to open multiple indexes? (since IndexReader will open
> >> one
> >> directory at a time) could any1 please suggest me?
> >> 
> >> 
> >> --
> >> View this message in context: http://www.nabble.com/index-reader-for-
> >> multiple-indexes-tp25716741p25716741.html
> >> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >> 
> >> 
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> > 
> > 
> 
> -- 
> View this message in context: http://www.nabble.com/index-reader-for-multiple-indexes-tp25716741p25726159.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

-- 
David Causse
Spotter
http://www.spotter.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Right we're getting the spans, however it's just the payloads that are
missing, randomly...

On Wed, Dec 9, 2009 at 2:23 AM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> There was a thread a while back about how span queries don't enumerate
> every possible span, but I can't remember if that included sometimes
> missing payloads...
>
> Mike
>
> On Tue, Dec 8, 2009 at 7:34 PM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
>> Howdy,
>>
>> I am wondering if anyone has seen
>> NearSpansUnordered.getPayload() not return payloads that are
>> verifiably accessible via IR.termPositions? It's a bit confusing
>> because most of the time they're returned properly.
>>
>> I suspect the payload logic gets tripped up in
>> NearSpansUnordered. I'll put together a test case, however the
>> difficulty is that we're only seeing the issue with largish 800
>> MB indexes, which could make the test case a little crazy.
>>
>> Jason
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

> Don't you have a playground to properly test your changes

Yes, I'll be doing a practice run in a DEV cluster. It is the practice run that I'm planning
at this point. 

Many thanks for your pointers, Danil. 

-----Original Message-----
From: Danil ŢORIN [mailto:torindan@gmail.com] 
Sent: 09 December 2009 16:03
To: java-user@lucene.apache.org
Subject: Re: Index file compatibility and a migration plan to lucene 3

There are a LOT of deprecated stuff in 2.9.1 (but it's still there)
and your code should run as it is
(however there are some changes in behavior, so read carefully CHANGES.txt)

In 3.0 this old stuff is removed.
Your production readers may not even start (which I guess is more
painful than 2 step transition)

The safest way is to upgrade to 2.9.1, fix ALL deprecation warnings,
and only then move on to 3.0.

Don't you have a playground to properly test your changes before
deploying to production ?

On Wed, Dec 9, 2009 at 17:55, Rob Staveley (Tom) <rstaveley@seseit.com> wrote:
> COMPRESS is supported (only deprecated) in 2.9.1, so I'm expecting them to be supported
http://lucene.apache.org/java/2_9_1/api/all/org/apache/lucene/document/Field.Store.html#COMPRESS
>
> I guess I should expect optimize() to increase the size of the index as compressed fields
are expanded as it says in item 2 at http://lucene.apache.org/java/3_0_0/changes/Changes.html#3.0.0.changes_in_runtime_behavior
>
> That must mean that 3.0.0 can read 2.9.1 indexes, but I'm wondering if I shouldn't simply
upgrade the readers straight from 2.3.1 to 3.0.0 now with an immediate optimize handling the
conversion. Can I safely assume that 3.0.0 is able to read 2.3.1?
>
> Making code changes to the readers in production is tricky in my infrastructure and making
one transition rather than two is very desirable.
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:torindan@gmail.com]
> Sent: 09 December 2009 15:15
> To: java-user@lucene.apache.org
> Subject: Re: Index file compatibility and a migration plan to lucene 3
>
> 2nd point can be simply archived by an optimize (which will read old
> segments and will create a new one)
> But I'm not sure how it handles compressed fields.
>
>
> On Wed, Dec 9, 2009 at 16:50, Rob Staveley (Tom) <rstaveley@seseit.com> wrote:
>> Thanks, Danil. I think you've saved me a lot of time. Weiwei too - converting rather
than reindexing everything, which will save a lot of time.
>>
>> So, I should do this:
>>
>> 1. Convert readers to 2.9.1, which should be able to read any 2.x index including
the existing 2.3.1 indexes
>> 2. Convert writers to 2.9.1, using Weiwei's idea (converting the index with a 2.9.1
reader+writer conversion utility) to save some time.
>> 3. Have the writers push converted indexes to the readers using the existing production
infrastructure
>> 4. Like (9.) in my original plan. [Go through my index writers and index reader clients
and systematically purge all of the Field.Store.COMPRESS fields and migrate to an explicit
CompressionTools approach where applicable and no compression where applicable. During this
phase I'll expect to have CompressionTools-compressed fields coexisting with their Field.Store.COMPRESS
predecessors, where index reader client use of Field.Store.COMPRESS is in transit to the explicit
decompression approach.]
>> 5. Convert the readers to 3.0.0, which should be able to read 2.9.1, if there are
no compressed fields (??)
>> 6. Convert the writers to 3.0.0
>>
>>
>>
>> -----Original Message-----
>> From: Danil ŢORIN [mailto:torindan@gmail.com]
>> Sent: 09 December 2009 13:20
>> To: java-user@lucene.apache.org
>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>
>> You NEED to update your readers first, or else they will be unable to
>> read files created by newer version.
>> And trust me, there are changes in index format from 2.3 -> 2.9
>>
>> On Wed, Dec 9, 2009 at 15:11, Weiwei Wang <ww.wang.cs@gmail.com> wrote:
>>> Hi, Rob,
>>> I read
>>> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
>>> found no compatibility guarantee for IndexWriter between different version.
>>>
>>> You can run your idea as a test and see the output.
>>> If it doesn't work, i suggest you convert your index to new version as I
>>> said in the last post.
>>>
>>> You can develop a convert tool to do this job automatically(that what i have
>>> done).
>>>
>>> If you do not have full access to the data center, you can read(readonly
>>> mode is preferred) from the data center(through nfs or something like that)
>>> and write to your local disk.
>>>
>>> When all converting is done, you can copy the new index to the data center
>>> with the help of the administrator.
>>>
>>> On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:
>>>
>>>> Thanks for the swift response, Weiwei.
>>>>
>>>> In my deployment, my index readers are in a data centre and therefore more
>>>> difficult to upgrade than the writers. That's why I wanted to start with
the
>>>> writers rather than the readers. I realise that it looks the wrong way round
>>>> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
>>>> situations, but I wanted to know if writer first would work for me for 2.3.1
>>>> -> 3.0.0.
>>>>
>>>> -----Original Message-----
>>>> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
>>>> Sent: 09 December 2009 12:21
>>>> To: java-user@lucene.apache.org
>>>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>>>
>>>> Iā€™ve finished a upgrade from 2.4.1 to 3.0.0
>>>>
>>>> What I do is like this:
>>>> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
>>>> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
>>>> 3.0.0 IndexWriter to write all the documents into a new index
>>>> 3. Update QueryPaser to 3.0.0
>>>>
>>>> I've redeployed my system and it works fine now.
>>>>
>>>>
>>>> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
>>>> >wrote:
>>>>
>>>> > I have Lucene 2.3.1 code and indexes deployed in production in a
>>>> > distributed
>>>> > system and would like to bring everything up to date with 3.0.0 via
>>>> 2.9.1.
>>>> >
>>>> > Here's my migration plan:
>>>> >
>>>> > 1. Add a index writer which generates a 2.9.1 "test" index
>>>> > 2. Have that "test" index writer push that 2.9.1 "test" index into
>>>> > production and see if the distributed 2.3.1 index readers can cope with
>>>> it
>>>> > OK
>>>> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
>>>> -
>>>> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See
if
>>>> it
>>>> > works.
>>>> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
>>>> > but with support for explicit use of CompressionTools for decompression,
>>>> > where fields have been explicitly compressed with CompressionTools -
the
>>>> > application will knows which need decompression)
>>>> > 5. Add a CompressionTools to my "test" index writer, generating
>>>> explicitly
>>>> > compressed fields in the 2.9.1 "test" index
>>>> > 6. Test explicit decompression for relevant fields with CompressionTools
>>>> in
>>>> > my 2.9.1 "test" index in my index readers
>>>> > 7. Upgrade my "test" index writer to 3.0.0
>>>> > 8. Have that "test" index writer push that 3.0.0 "test" index into
>>>> > production and see if the distributed 2.9.1 index readers can cope with
>>>> it
>>>> > OK
>>>> > 9. Go through my index writers and index reader clients and
>>>> systematically
>>>> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
>>>> > CompressionTools approach where applicable and no compression where
>>>> > applicable. During this phase I'll expect to have
>>>> > CompressionTools-compressed fields coexisting with their
>>>> > Field.Store.COMPRESS predecessors, where index reader client use of
>>>> > Field.Store.COMPRESS is in transit to the explicit decompression
>>>> approach.
>>>> > 10. Upgrade my index writers to 3.0.0
>>>> > 11. Upgrade my index readers to 3.0.0
>>>> >
>>>> > I've simplified this a bit, because I shan't really be testing straight
>>>> off
>>>> > in production(!) - I'll test the migration plan in a test cluster first;
>>>> > but
>>>> > this gives you the idea about the path.
>>>> >
>>>> > I wanted to know if I should expect problems with this plan. I'm
>>>> depending
>>>> > on newer writers generating indexes for older readers and 3 is a major
>>>> > number upgrade. It looks like I can get away with this in version 3,
but
>>>> > that's by no means a guarantee according to
>>>> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>>>> >
>>>> > Does this sound like a good plan?
>>>> >
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> Weiwei Wang
>>>> Alex Wang
>>>> ēˇ‹å·¨å·¨
>>>> Room 403, Mengmin Wei Building
>>>> Computer Science Department
>>>> Gulou Campus of Nanjing University
>>>> Nanjing, P.R.China, 210093
>>>>
>>>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Weiwei Wang
>>> Alex Wang
>>> ēˇ‹å·¨å·¨
>>> Room 403, Mengmin Wei Building
>>> Computer Science Department
>>> Gulou Campus of Nanjing University
>>> Nanjing, P.R.China, 210093
>>>
>>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

There are a LOT of deprecated stuff in 2.9.1 (but it's still there)
and your code should run as it is
(however there are some changes in behavior, so read carefully CHANGES.txt)

In 3.0 this old stuff is removed.
Your production readers may not even start (which I guess is more
painful than 2 step transition)

The safest way is to upgrade to 2.9.1, fix ALL deprecation warnings,
and only then move on to 3.0.

Don't you have a playground to properly test your changes before
deploying to production ?

On Wed, Dec 9, 2009 at 17:55, Rob Staveley (Tom) <rstaveley@seseit.com> wrote:
> COMPRESS is supported (only deprecated) in 2.9.1, so I'm expecting them to be supported
http://lucene.apache.org/java/2_9_1/api/all/org/apache/lucene/document/Field.Store.html#COMPRESS
>
> I guess I should expect optimize() to increase the size of the index as compressed fields
are expanded as it says in item 2 at http://lucene.apache.org/java/3_0_0/changes/Changes.html#3.0.0.changes_in_runtime_behavior
>
> That must mean that 3.0.0 can read 2.9.1 indexes, but I'm wondering if I shouldn't simply
upgrade the readers straight from 2.3.1 to 3.0.0 now with an immediate optimize handling the
conversion. Can I safely assume that 3.0.0 is able to read 2.3.1?
>
> Making code changes to the readers in production is tricky in my infrastructure and making
one transition rather than two is very desirable.
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:torindan@gmail.com]
> Sent: 09 December 2009 15:15
> To: java-user@lucene.apache.org
> Subject: Re: Index file compatibility and a migration plan to lucene 3
>
> 2nd point can be simply archived by an optimize (which will read old
> segments and will create a new one)
> But I'm not sure how it handles compressed fields.
>
>
> On Wed, Dec 9, 2009 at 16:50, Rob Staveley (Tom) <rstaveley@seseit.com> wrote:
>> Thanks, Danil. I think you've saved me a lot of time. Weiwei too - converting rather
than reindexing everything, which will save a lot of time.
>>
>> So, I should do this:
>>
>> 1. Convert readers to 2.9.1, which should be able to read any 2.x index including
the existing 2.3.1 indexes
>> 2. Convert writers to 2.9.1, using Weiwei's idea (converting the index with a 2.9.1
reader+writer conversion utility) to save some time.
>> 3. Have the writers push converted indexes to the readers using the existing production
infrastructure
>> 4. Like (9.) in my original plan. [Go through my index writers and index reader clients
and systematically purge all of the Field.Store.COMPRESS fields and migrate to an explicit
CompressionTools approach where applicable and no compression where applicable. During this
phase I'll expect to have CompressionTools-compressed fields coexisting with their Field.Store.COMPRESS
predecessors, where index reader client use of Field.Store.COMPRESS is in transit to the explicit
decompression approach.]
>> 5. Convert the readers to 3.0.0, which should be able to read 2.9.1, if there are
no compressed fields (??)
>> 6. Convert the writers to 3.0.0
>>
>>
>>
>> -----Original Message-----
>> From: Danil ŢORIN [mailto:torindan@gmail.com]
>> Sent: 09 December 2009 13:20
>> To: java-user@lucene.apache.org
>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>
>> You NEED to update your readers first, or else they will be unable to
>> read files created by newer version.
>> And trust me, there are changes in index format from 2.3 -> 2.9
>>
>> On Wed, Dec 9, 2009 at 15:11, Weiwei Wang <ww.wang.cs@gmail.com> wrote:
>>> Hi, Rob,
>>> I read
>>> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
>>> found no compatibility guarantee for IndexWriter between different version.
>>>
>>> You can run your idea as a test and see the output.
>>> If it doesn't work, i suggest you convert your index to new version as I
>>> said in the last post.
>>>
>>> You can develop a convert tool to do this job automatically(that what i have
>>> done).
>>>
>>> If you do not have full access to the data center, you can read(readonly
>>> mode is preferred) from the data center(through nfs or something like that)
>>> and write to your local disk.
>>>
>>> When all converting is done, you can copy the new index to the data center
>>> with the help of the administrator.
>>>
>>> On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:
>>>
>>>> Thanks for the swift response, Weiwei.
>>>>
>>>> In my deployment, my index readers are in a data centre and therefore more
>>>> difficult to upgrade than the writers. That's why I wanted to start with
the
>>>> writers rather than the readers. I realise that it looks the wrong way round
>>>> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
>>>> situations, but I wanted to know if writer first would work for me for 2.3.1
>>>> -> 3.0.0.
>>>>
>>>> -----Original Message-----
>>>> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
>>>> Sent: 09 December 2009 12:21
>>>> To: java-user@lucene.apache.org
>>>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>>>
>>>> Iā€™ve finished a upgrade from 2.4.1 to 3.0.0
>>>>
>>>> What I do is like this:
>>>> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
>>>> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
>>>> 3.0.0 IndexWriter to write all the documents into a new index
>>>> 3. Update QueryPaser to 3.0.0
>>>>
>>>> I've redeployed my system and it works fine now.
>>>>
>>>>
>>>> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
>>>> >wrote:
>>>>
>>>> > I have Lucene 2.3.1 code and indexes deployed in production in a
>>>> > distributed
>>>> > system and would like to bring everything up to date with 3.0.0 via
>>>> 2.9.1.
>>>> >
>>>> > Here's my migration plan:
>>>> >
>>>> > 1. Add a index writer which generates a 2.9.1 "test" index
>>>> > 2. Have that "test" index writer push that 2.9.1 "test" index into
>>>> > production and see if the distributed 2.3.1 index readers can cope with
>>>> it
>>>> > OK
>>>> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
>>>> -
>>>> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See
if
>>>> it
>>>> > works.
>>>> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
>>>> > but with support for explicit use of CompressionTools for decompression,
>>>> > where fields have been explicitly compressed with CompressionTools -
the
>>>> > application will knows which need decompression)
>>>> > 5. Add a CompressionTools to my "test" index writer, generating
>>>> explicitly
>>>> > compressed fields in the 2.9.1 "test" index
>>>> > 6. Test explicit decompression for relevant fields with CompressionTools
>>>> in
>>>> > my 2.9.1 "test" index in my index readers
>>>> > 7. Upgrade my "test" index writer to 3.0.0
>>>> > 8. Have that "test" index writer push that 3.0.0 "test" index into
>>>> > production and see if the distributed 2.9.1 index readers can cope with
>>>> it
>>>> > OK
>>>> > 9. Go through my index writers and index reader clients and
>>>> systematically
>>>> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
>>>> > CompressionTools approach where applicable and no compression where
>>>> > applicable. During this phase I'll expect to have
>>>> > CompressionTools-compressed fields coexisting with their
>>>> > Field.Store.COMPRESS predecessors, where index reader client use of
>>>> > Field.Store.COMPRESS is in transit to the explicit decompression
>>>> approach.
>>>> > 10. Upgrade my index writers to 3.0.0
>>>> > 11. Upgrade my index readers to 3.0.0
>>>> >
>>>> > I've simplified this a bit, because I shan't really be testing straight
>>>> off
>>>> > in production(!) - I'll test the migration plan in a test cluster first;
>>>> > but
>>>> > this gives you the idea about the path.
>>>> >
>>>> > I wanted to know if I should expect problems with this plan. I'm
>>>> depending
>>>> > on newer writers generating indexes for older readers and 3 is a major
>>>> > number upgrade. It looks like I can get away with this in version 3,
but
>>>> > that's by no means a guarantee according to
>>>> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>>>> >
>>>> > Does this sound like a good plan?
>>>> >
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> Weiwei Wang
>>>> Alex Wang
>>>> ēˇ‹å·¨å·¨
>>>> Room 403, Mengmin Wei Building
>>>> Computer Science Department
>>>> Gulou Campus of Nanjing University
>>>> Nanjing, P.R.China, 210093
>>>>
>>>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Weiwei Wang
>>> Alex Wang
>>> ēˇ‹å·¨å·¨
>>> Room 403, Mengmin Wei Building
>>> Computer Science Department
>>> Gulou Campus of Nanjing University
>>> Nanjing, P.R.China, 210093
>>>
>>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

COMPRESS is supported (only deprecated) in 2.9.1, so I'm expecting them to be supported http://lucene.apache.org/java/2_9_1/api/all/org/apache/lucene/document/Field.Store.html#COMPRESS

I guess I should expect optimize() to increase the size of the index as compressed fields
are expanded as it says in item 2 at http://lucene.apache.org/java/3_0_0/changes/Changes.html#3.0.0.changes_in_runtime_behavior


That must mean that 3.0.0 can read 2.9.1 indexes, but I'm wondering if I shouldn't simply
upgrade the readers straight from 2.3.1 to 3.0.0 now with an immediate optimize handling the
conversion. Can I safely assume that 3.0.0 is able to read 2.3.1?

Making code changes to the readers in production is tricky in my infrastructure and making
one transition rather than two is very desirable.

-----Original Message-----
From: Danil ŢORIN [mailto:torindan@gmail.com] 
Sent: 09 December 2009 15:15
To: java-user@lucene.apache.org
Subject: Re: Index file compatibility and a migration plan to lucene 3

2nd point can be simply archived by an optimize (which will read old
segments and will create a new one)
But I'm not sure how it handles compressed fields.


On Wed, Dec 9, 2009 at 16:50, Rob Staveley (Tom) <rstaveley@seseit.com> wrote:
> Thanks, Danil. I think you've saved me a lot of time. Weiwei too - converting rather
than reindexing everything, which will save a lot of time.
>
> So, I should do this:
>
> 1. Convert readers to 2.9.1, which should be able to read any 2.x index including the
existing 2.3.1 indexes
> 2. Convert writers to 2.9.1, using Weiwei's idea (converting the index with a 2.9.1 reader+writer
conversion utility) to save some time.
> 3. Have the writers push converted indexes to the readers using the existing production
infrastructure
> 4. Like (9.) in my original plan. [Go through my index writers and index reader clients
and systematically purge all of the Field.Store.COMPRESS fields and migrate to an explicit
CompressionTools approach where applicable and no compression where applicable. During this
phase I'll expect to have CompressionTools-compressed fields coexisting with their Field.Store.COMPRESS
predecessors, where index reader client use of Field.Store.COMPRESS is in transit to the explicit
decompression approach.]
> 5. Convert the readers to 3.0.0, which should be able to read 2.9.1, if there are no
compressed fields (??)
> 6. Convert the writers to 3.0.0
>
>
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:torindan@gmail.com]
> Sent: 09 December 2009 13:20
> To: java-user@lucene.apache.org
> Subject: Re: Index file compatibility and a migration plan to lucene 3
>
> You NEED to update your readers first, or else they will be unable to
> read files created by newer version.
> And trust me, there are changes in index format from 2.3 -> 2.9
>
> On Wed, Dec 9, 2009 at 15:11, Weiwei Wang <ww.wang.cs@gmail.com> wrote:
>> Hi, Rob,
>> I read
>> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
>> found no compatibility guarantee for IndexWriter between different version.
>>
>> You can run your idea as a test and see the output.
>> If it doesn't work, i suggest you convert your index to new version as I
>> said in the last post.
>>
>> You can develop a convert tool to do this job automatically(that what i have
>> done).
>>
>> If you do not have full access to the data center, you can read(readonly
>> mode is preferred) from the data center(through nfs or something like that)
>> and write to your local disk.
>>
>> When all converting is done, you can copy the new index to the data center
>> with the help of the administrator.
>>
>> On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:
>>
>>> Thanks for the swift response, Weiwei.
>>>
>>> In my deployment, my index readers are in a data centre and therefore more
>>> difficult to upgrade than the writers. That's why I wanted to start with the
>>> writers rather than the readers. I realise that it looks the wrong way round
>>> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
>>> situations, but I wanted to know if writer first would work for me for 2.3.1
>>> -> 3.0.0.
>>>
>>> -----Original Message-----
>>> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
>>> Sent: 09 December 2009 12:21
>>> To: java-user@lucene.apache.org
>>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>>
>>> Iā€™ve finished a upgrade from 2.4.1 to 3.0.0
>>>
>>> What I do is like this:
>>> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
>>> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
>>> 3.0.0 IndexWriter to write all the documents into a new index
>>> 3. Update QueryPaser to 3.0.0
>>>
>>> I've redeployed my system and it works fine now.
>>>
>>>
>>> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
>>> >wrote:
>>>
>>> > I have Lucene 2.3.1 code and indexes deployed in production in a
>>> > distributed
>>> > system and would like to bring everything up to date with 3.0.0 via
>>> 2.9.1.
>>> >
>>> > Here's my migration plan:
>>> >
>>> > 1. Add a index writer which generates a 2.9.1 "test" index
>>> > 2. Have that "test" index writer push that 2.9.1 "test" index into
>>> > production and see if the distributed 2.3.1 index readers can cope with
>>> it
>>> > OK
>>> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
>>> -
>>> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if
>>> it
>>> > works.
>>> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
>>> > but with support for explicit use of CompressionTools for decompression,
>>> > where fields have been explicitly compressed with CompressionTools - the
>>> > application will knows which need decompression)
>>> > 5. Add a CompressionTools to my "test" index writer, generating
>>> explicitly
>>> > compressed fields in the 2.9.1 "test" index
>>> > 6. Test explicit decompression for relevant fields with CompressionTools
>>> in
>>> > my 2.9.1 "test" index in my index readers
>>> > 7. Upgrade my "test" index writer to 3.0.0
>>> > 8. Have that "test" index writer push that 3.0.0 "test" index into
>>> > production and see if the distributed 2.9.1 index readers can cope with
>>> it
>>> > OK
>>> > 9. Go through my index writers and index reader clients and
>>> systematically
>>> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
>>> > CompressionTools approach where applicable and no compression where
>>> > applicable. During this phase I'll expect to have
>>> > CompressionTools-compressed fields coexisting with their
>>> > Field.Store.COMPRESS predecessors, where index reader client use of
>>> > Field.Store.COMPRESS is in transit to the explicit decompression
>>> approach.
>>> > 10. Upgrade my index writers to 3.0.0
>>> > 11. Upgrade my index readers to 3.0.0
>>> >
>>> > I've simplified this a bit, because I shan't really be testing straight
>>> off
>>> > in production(!) - I'll test the migration plan in a test cluster first;
>>> > but
>>> > this gives you the idea about the path.
>>> >
>>> > I wanted to know if I should expect problems with this plan. I'm
>>> depending
>>> > on newer writers generating indexes for older readers and 3 is a major
>>> > number upgrade. It looks like I can get away with this in version 3, but
>>> > that's by no means a guarantee according to
>>> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>>> >
>>> > Does this sound like a good plan?
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >
>>> >
>>>
>>>
>>> --
>>> Weiwei Wang
>>> Alex Wang
>>> ēˇ‹å·¨å·¨
>>> Room 403, Mengmin Wei Building
>>> Computer Science Department
>>> Gulou Campus of Nanjing University
>>> Nanjing, P.R.China, 210093
>>>
>>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>> --
>> Weiwei Wang
>> Alex Wang
>> ēˇ‹å·¨å·¨
>> Room 403, Mengmin Wei Building
>> Computer Science Department
>> Gulou Campus of Nanjing University
>> Nanjing, P.R.China, 210093
>>
>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

2nd point can be simply archived by an optimize (which will read old
segments and will create a new one)
But I'm not sure how it handles compressed fields.


On Wed, Dec 9, 2009 at 16:50, Rob Staveley (Tom) <rstaveley@seseit.com> wrote:
> Thanks, Danil. I think you've saved me a lot of time. Weiwei too - converting rather
than reindexing everything, which will save a lot of time.
>
> So, I should do this:
>
> 1. Convert readers to 2.9.1, which should be able to read any 2.x index including the
existing 2.3.1 indexes
> 2. Convert writers to 2.9.1, using Weiwei's idea (converting the index with a 2.9.1 reader+writer
conversion utility) to save some time.
> 3. Have the writers push converted indexes to the readers using the existing production
infrastructure
> 4. Like (9.) in my original plan. [Go through my index writers and index reader clients
and systematically purge all of the Field.Store.COMPRESS fields and migrate to an explicit
CompressionTools approach where applicable and no compression where applicable. During this
phase I'll expect to have CompressionTools-compressed fields coexisting with their Field.Store.COMPRESS
predecessors, where index reader client use of Field.Store.COMPRESS is in transit to the explicit
decompression approach.]
> 5. Convert the readers to 3.0.0, which should be able to read 2.9.1, if there are no
compressed fields (??)
> 6. Convert the writers to 3.0.0
>
>
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:torindan@gmail.com]
> Sent: 09 December 2009 13:20
> To: java-user@lucene.apache.org
> Subject: Re: Index file compatibility and a migration plan to lucene 3
>
> You NEED to update your readers first, or else they will be unable to
> read files created by newer version.
> And trust me, there are changes in index format from 2.3 -> 2.9
>
> On Wed, Dec 9, 2009 at 15:11, Weiwei Wang <ww.wang.cs@gmail.com> wrote:
>> Hi, Rob,
>> I read
>> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
>> found no compatibility guarantee for IndexWriter between different version.
>>
>> You can run your idea as a test and see the output.
>> If it doesn't work, i suggest you convert your index to new version as I
>> said in the last post.
>>
>> You can develop a convert tool to do this job automatically(that what i have
>> done).
>>
>> If you do not have full access to the data center, you can read(readonly
>> mode is preferred) from the data center(through nfs or something like that)
>> and write to your local disk.
>>
>> When all converting is done, you can copy the new index to the data center
>> with the help of the administrator.
>>
>> On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:
>>
>>> Thanks for the swift response, Weiwei.
>>>
>>> In my deployment, my index readers are in a data centre and therefore more
>>> difficult to upgrade than the writers. That's why I wanted to start with the
>>> writers rather than the readers. I realise that it looks the wrong way round
>>> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
>>> situations, but I wanted to know if writer first would work for me for 2.3.1
>>> -> 3.0.0.
>>>
>>> -----Original Message-----
>>> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
>>> Sent: 09 December 2009 12:21
>>> To: java-user@lucene.apache.org
>>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>>
>>> Iā€™ve finished a upgrade from 2.4.1 to 3.0.0
>>>
>>> What I do is like this:
>>> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
>>> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
>>> 3.0.0 IndexWriter to write all the documents into a new index
>>> 3. Update QueryPaser to 3.0.0
>>>
>>> I've redeployed my system and it works fine now.
>>>
>>>
>>> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
>>> >wrote:
>>>
>>> > I have Lucene 2.3.1 code and indexes deployed in production in a
>>> > distributed
>>> > system and would like to bring everything up to date with 3.0.0 via
>>> 2.9.1.
>>> >
>>> > Here's my migration plan:
>>> >
>>> > 1. Add a index writer which generates a 2.9.1 "test" index
>>> > 2. Have that "test" index writer push that 2.9.1 "test" index into
>>> > production and see if the distributed 2.3.1 index readers can cope with
>>> it
>>> > OK
>>> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
>>> -
>>> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if
>>> it
>>> > works.
>>> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
>>> > but with support for explicit use of CompressionTools for decompression,
>>> > where fields have been explicitly compressed with CompressionTools - the
>>> > application will knows which need decompression)
>>> > 5. Add a CompressionTools to my "test" index writer, generating
>>> explicitly
>>> > compressed fields in the 2.9.1 "test" index
>>> > 6. Test explicit decompression for relevant fields with CompressionTools
>>> in
>>> > my 2.9.1 "test" index in my index readers
>>> > 7. Upgrade my "test" index writer to 3.0.0
>>> > 8. Have that "test" index writer push that 3.0.0 "test" index into
>>> > production and see if the distributed 2.9.1 index readers can cope with
>>> it
>>> > OK
>>> > 9. Go through my index writers and index reader clients and
>>> systematically
>>> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
>>> > CompressionTools approach where applicable and no compression where
>>> > applicable. During this phase I'll expect to have
>>> > CompressionTools-compressed fields coexisting with their
>>> > Field.Store.COMPRESS predecessors, where index reader client use of
>>> > Field.Store.COMPRESS is in transit to the explicit decompression
>>> approach.
>>> > 10. Upgrade my index writers to 3.0.0
>>> > 11. Upgrade my index readers to 3.0.0
>>> >
>>> > I've simplified this a bit, because I shan't really be testing straight
>>> off
>>> > in production(!) - I'll test the migration plan in a test cluster first;
>>> > but
>>> > this gives you the idea about the path.
>>> >
>>> > I wanted to know if I should expect problems with this plan. I'm
>>> depending
>>> > on newer writers generating indexes for older readers and 3 is a major
>>> > number upgrade. It looks like I can get away with this in version 3, but
>>> > that's by no means a guarantee according to
>>> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>>> >
>>> > Does this sound like a good plan?
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >
>>> >
>>>
>>>
>>> --
>>> Weiwei Wang
>>> Alex Wang
>>> ēˇ‹å·¨å·¨
>>> Room 403, Mengmin Wei Building
>>> Computer Science Department
>>> Gulou Campus of Nanjing University
>>> Nanjing, P.R.China, 210093
>>>
>>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>> --
>> Weiwei Wang
>> Alex Wang
>> ēˇ‹å·¨å·¨
>> Room 403, Mengmin Wei Building
>> Computer Science Department
>> Gulou Campus of Nanjing University
>> Nanjing, P.R.China, 210093
>>
>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Thanks, Danil. I think you've saved me a lot of time. Weiwei too - converting rather than reindexing
everything, which will save a lot of time.

So, I should do this:

1. Convert readers to 2.9.1, which should be able to read any 2.x index including the existing
2.3.1 indexes
2. Convert writers to 2.9.1, using Weiwei's idea (converting the index with a 2.9.1 reader+writer
conversion utility) to save some time.
3. Have the writers push converted indexes to the readers using the existing production infrastructure
4. Like (9.) in my original plan. [Go through my index writers and index reader clients and
systematically purge all of the Field.Store.COMPRESS fields and migrate to an explicit CompressionTools
approach where applicable and no compression where applicable. During this phase I'll expect
to have CompressionTools-compressed fields coexisting with their Field.Store.COMPRESS predecessors,
where index reader client use of Field.Store.COMPRESS is in transit to the explicit decompression
approach.]
5. Convert the readers to 3.0.0, which should be able to read 2.9.1, if there are no compressed
fields (??)
6. Convert the writers to 3.0.0



-----Original Message-----
From: Danil ŢORIN [mailto:torindan@gmail.com] 
Sent: 09 December 2009 13:20
To: java-user@lucene.apache.org
Subject: Re: Index file compatibility and a migration plan to lucene 3

You NEED to update your readers first, or else they will be unable to
read files created by newer version.
And trust me, there are changes in index format from 2.3 -> 2.9

On Wed, Dec 9, 2009 at 15:11, Weiwei Wang <ww.wang.cs@gmail.com> wrote:
> Hi, Rob,
> I read
> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
> found no compatibility guarantee for IndexWriter between different version.
>
> You can run your idea as a test and see the output.
> If it doesn't work, i suggest you convert your index to new version as I
> said in the last post.
>
> You can develop a convert tool to do this job automatically(that what i have
> done).
>
> If you do not have full access to the data center, you can read(readonly
> mode is preferred) from the data center(through nfs or something like that)
> and write to your local disk.
>
> When all converting is done, you can copy the new index to the data center
> with the help of the administrator.
>
> On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:
>
>> Thanks for the swift response, Weiwei.
>>
>> In my deployment, my index readers are in a data centre and therefore more
>> difficult to upgrade than the writers. That's why I wanted to start with the
>> writers rather than the readers. I realise that it looks the wrong way round
>> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
>> situations, but I wanted to know if writer first would work for me for 2.3.1
>> -> 3.0.0.
>>
>> -----Original Message-----
>> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
>> Sent: 09 December 2009 12:21
>> To: java-user@lucene.apache.org
>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>
>> Iā€™ve finished a upgrade from 2.4.1 to 3.0.0
>>
>> What I do is like this:
>> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
>> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
>> 3.0.0 IndexWriter to write all the documents into a new index
>> 3. Update QueryPaser to 3.0.0
>>
>> I've redeployed my system and it works fine now.
>>
>>
>> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
>> >wrote:
>>
>> > I have Lucene 2.3.1 code and indexes deployed in production in a
>> > distributed
>> > system and would like to bring everything up to date with 3.0.0 via
>> 2.9.1.
>> >
>> > Here's my migration plan:
>> >
>> > 1. Add a index writer which generates a 2.9.1 "test" index
>> > 2. Have that "test" index writer push that 2.9.1 "test" index into
>> > production and see if the distributed 2.3.1 index readers can cope with
>> it
>> > OK
>> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
>> -
>> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if
>> it
>> > works.
>> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
>> > but with support for explicit use of CompressionTools for decompression,
>> > where fields have been explicitly compressed with CompressionTools - the
>> > application will knows which need decompression)
>> > 5. Add a CompressionTools to my "test" index writer, generating
>> explicitly
>> > compressed fields in the 2.9.1 "test" index
>> > 6. Test explicit decompression for relevant fields with CompressionTools
>> in
>> > my 2.9.1 "test" index in my index readers
>> > 7. Upgrade my "test" index writer to 3.0.0
>> > 8. Have that "test" index writer push that 3.0.0 "test" index into
>> > production and see if the distributed 2.9.1 index readers can cope with
>> it
>> > OK
>> > 9. Go through my index writers and index reader clients and
>> systematically
>> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
>> > CompressionTools approach where applicable and no compression where
>> > applicable. During this phase I'll expect to have
>> > CompressionTools-compressed fields coexisting with their
>> > Field.Store.COMPRESS predecessors, where index reader client use of
>> > Field.Store.COMPRESS is in transit to the explicit decompression
>> approach.
>> > 10. Upgrade my index writers to 3.0.0
>> > 11. Upgrade my index readers to 3.0.0
>> >
>> > I've simplified this a bit, because I shan't really be testing straight
>> off
>> > in production(!) - I'll test the migration plan in a test cluster first;
>> > but
>> > this gives you the idea about the path.
>> >
>> > I wanted to know if I should expect problems with this plan. I'm
>> depending
>> > on newer writers generating indexes for older readers and 3 is a major
>> > number upgrade. It looks like I can get away with this in version 3, but
>> > that's by no means a guarantee according to
>> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>> >
>> > Does this sound like a good plan?
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>>
>>
>> --
>> Weiwei Wang
>> Alex Wang
>> ēˇ‹å·¨å·¨
>> Room 403, Mengmin Wei Building
>> Computer Science Department
>> Gulou Campus of Nanjing University
>> Nanjing, P.R.China, 210093
>>
>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Weiwei Wang
> Alex Wang
> ēˇ‹å·¨å·¨
> Room 403, Mengmin Wei Building
> Computer Science Department
> Gulou Campus of Nanjing University
> Nanjing, P.R.China, 210093
>
> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

You NEED to update your readers first, or else they will be unable to
read files created by newer version.
And trust me, there are changes in index format from 2.3 -> 2.9

On Wed, Dec 9, 2009 at 15:11, Weiwei Wang <ww.wang.cs@gmail.com> wrote:
> Hi, Rob,
> I read
> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
> found no compatibility guarantee for IndexWriter between different version.
>
> You can run your idea as a test and see the output.
> If it doesn't work, i suggest you convert your index to new version as I
> said in the last post.
>
> You can develop a convert tool to do this job automatically(that what i have
> done).
>
> If you do not have full access to the data center, you can read(readonly
> mode is preferred) from the data center(through nfs or something like that)
> and write to your local disk.
>
> When all converting is done, you can copy the new index to the data center
> with the help of the administrator.
>
> On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:
>
>> Thanks for the swift response, Weiwei.
>>
>> In my deployment, my index readers are in a data centre and therefore more
>> difficult to upgrade than the writers. That's why I wanted to start with the
>> writers rather than the readers. I realise that it looks the wrong way round
>> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
>> situations, but I wanted to know if writer first would work for me for 2.3.1
>> -> 3.0.0.
>>
>> -----Original Message-----
>> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
>> Sent: 09 December 2009 12:21
>> To: java-user@lucene.apache.org
>> Subject: Re: Index file compatibility and a migration plan to lucene 3
>>
>> Iā€™ve finished a upgrade from 2.4.1 to 3.0.0
>>
>> What I do is like this:
>> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
>> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
>> 3.0.0 IndexWriter to write all the documents into a new index
>> 3. Update QueryPaser to 3.0.0
>>
>> I've redeployed my system and it works fine now.
>>
>>
>> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
>> >wrote:
>>
>> > I have Lucene 2.3.1 code and indexes deployed in production in a
>> > distributed
>> > system and would like to bring everything up to date with 3.0.0 via
>> 2.9.1.
>> >
>> > Here's my migration plan:
>> >
>> > 1. Add a index writer which generates a 2.9.1 "test" index
>> > 2. Have that "test" index writer push that 2.9.1 "test" index into
>> > production and see if the distributed 2.3.1 index readers can cope with
>> it
>> > OK
>> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
>> -
>> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if
>> it
>> > works.
>> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
>> > but with support for explicit use of CompressionTools for decompression,
>> > where fields have been explicitly compressed with CompressionTools - the
>> > application will knows which need decompression)
>> > 5. Add a CompressionTools to my "test" index writer, generating
>> explicitly
>> > compressed fields in the 2.9.1 "test" index
>> > 6. Test explicit decompression for relevant fields with CompressionTools
>> in
>> > my 2.9.1 "test" index in my index readers
>> > 7. Upgrade my "test" index writer to 3.0.0
>> > 8. Have that "test" index writer push that 3.0.0 "test" index into
>> > production and see if the distributed 2.9.1 index readers can cope with
>> it
>> > OK
>> > 9. Go through my index writers and index reader clients and
>> systematically
>> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
>> > CompressionTools approach where applicable and no compression where
>> > applicable. During this phase I'll expect to have
>> > CompressionTools-compressed fields coexisting with their
>> > Field.Store.COMPRESS predecessors, where index reader client use of
>> > Field.Store.COMPRESS is in transit to the explicit decompression
>> approach.
>> > 10. Upgrade my index writers to 3.0.0
>> > 11. Upgrade my index readers to 3.0.0
>> >
>> > I've simplified this a bit, because I shan't really be testing straight
>> off
>> > in production(!) - I'll test the migration plan in a test cluster first;
>> > but
>> > this gives you the idea about the path.
>> >
>> > I wanted to know if I should expect problems with this plan. I'm
>> depending
>> > on newer writers generating indexes for older readers and 3 is a major
>> > number upgrade. It looks like I can get away with this in version 3, but
>> > that's by no means a guarantee according to
>> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>> >
>> > Does this sound like a good plan?
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>>
>>
>> --
>> Weiwei Wang
>> Alex Wang
>> ēˇ‹å·¨å·¨
>> Room 403, Mengmin Wei Building
>> Computer Science Department
>> Gulou Campus of Nanjing University
>> Nanjing, P.R.China, 210093
>>
>> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Weiwei Wang
> Alex Wang
> ēˇ‹å·¨å·¨
> Room 403, Mengmin Wei Building
> Computer Science Department
> Gulou Campus of Nanjing University
> Nanjing, P.R.China, 210093
>
> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hi, Rob,
I read
http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
found no compatibility guarantee for IndexWriter between different version.

You can run your idea as a test and see the output.
If it doesn't work, i suggest you convert your index to new version as I
said in the last post.

You can develop a convert tool to do this job automatically(that what i have
done).

If you do not have full access to the data center, you can read(readonly
mode is preferred) from the data center(through nfs or something like that)
and write to your local disk.

When all converting is done, you can copy the new index to the data center
with the help of the administrator.

On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:

> Thanks for the swift response, Weiwei.
>
> In my deployment, my index readers are in a data centre and therefore more
> difficult to upgrade than the writers. That's why I wanted to start with the
> writers rather than the readers. I realise that it looks the wrong way round
> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
> situations, but I wanted to know if writer first would work for me for 2.3.1
> -> 3.0.0.
>
> -----Original Message-----
> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
> Sent: 09 December 2009 12:21
> To: java-user@lucene.apache.org
> Subject: Re: Index file compatibility and a migration plan to lucene 3
>
> Iā€™ve finished a upgrade from 2.4.1 to 3.0.0
>
> What I do is like this:
> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
> 3.0.0 IndexWriter to write all the documents into a new index
> 3. Update QueryPaser to 3.0.0
>
> I've redeployed my system and it works fine now.
>
>
> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
> >wrote:
>
> > I have Lucene 2.3.1 code and indexes deployed in production in a
> > distributed
> > system and would like to bring everything up to date with 3.0.0 via
> 2.9.1.
> >
> > Here's my migration plan:
> >
> > 1. Add a index writer which generates a 2.9.1 "test" index
> > 2. Have that "test" index writer push that 2.9.1 "test" index into
> > production and see if the distributed 2.3.1 index readers can cope with
> it
> > OK
> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
> -
> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if
> it
> > works.
> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
> > but with support for explicit use of CompressionTools for decompression,
> > where fields have been explicitly compressed with CompressionTools - the
> > application will knows which need decompression)
> > 5. Add a CompressionTools to my "test" index writer, generating
> explicitly
> > compressed fields in the 2.9.1 "test" index
> > 6. Test explicit decompression for relevant fields with CompressionTools
> in
> > my 2.9.1 "test" index in my index readers
> > 7. Upgrade my "test" index writer to 3.0.0
> > 8. Have that "test" index writer push that 3.0.0 "test" index into
> > production and see if the distributed 2.9.1 index readers can cope with
> it
> > OK
> > 9. Go through my index writers and index reader clients and
> systematically
> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
> > CompressionTools approach where applicable and no compression where
> > applicable. During this phase I'll expect to have
> > CompressionTools-compressed fields coexisting with their
> > Field.Store.COMPRESS predecessors, where index reader client use of
> > Field.Store.COMPRESS is in transit to the explicit decompression
> approach.
> > 10. Upgrade my index writers to 3.0.0
> > 11. Upgrade my index readers to 3.0.0
> >
> > I've simplified this a bit, because I shan't really be testing straight
> off
> > in production(!) - I'll test the migration plan in a test cluster first;
> > but
> > this gives you the idea about the path.
> >
> > I wanted to know if I should expect problems with this plan. I'm
> depending
> > on newer writers generating indexes for older readers and 3 is a major
> > number upgrade. It looks like I can get away with this in version 3, but
> > that's by no means a guarantee according to
> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
> >
> > Does this sound like a good plan?
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> Weiwei Wang
> Alex Wang
> ēˇ‹å·¨å·¨
> Room 403, Mengmin Wei Building
> Computer Science Department
> Gulou Campus of Nanjing University
> Nanjing, P.R.China, 210093
>
> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Weiwei Wang
Alex Wang
ēˇ‹å·¨å·¨
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

Thanks for the swift response, Weiwei.

In my deployment, my index readers are in a data centre and therefore more difficult to upgrade
than the writers. That's why I wanted to start with the writers rather than the readers. I
realise that it looks the wrong way round and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
effectively says that changing the reader first is a better idea for most situations, but
I wanted to know if writer first would work for me for 2.3.1 -> 3.0.0.

-----Original Message-----
From: Weiwei Wang [mailto:ww.wang.cs@gmail.com] 
Sent: 09 December 2009 12:21
To: java-user@lucene.apache.org
Subject: Re: Index file compatibility and a migration plan to lucene 3

Iā€™ve finished a upgrade from 2.4.1 to 3.0.0

What I do is like this:
1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
2. Use a 3.0.0 IndexReader to read the old version index and then use a
3.0.0 IndexWriter to write all the documents into a new index
3. Update QueryPaser to 3.0.0

I've redeployed my system and it works fine now.


On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:

> I have Lucene 2.3.1 code and indexes deployed in production in a
> distributed
> system and would like to bring everything up to date with 3.0.0 via 2.9.1.
>
> Here's my migration plan:
>
> 1. Add a index writer which generates a 2.9.1 "test" index
> 2. Have that "test" index writer push that 2.9.1 "test" index into
> production and see if the distributed 2.3.1 index readers can cope with it
> OK
> 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields) -
> we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if it
> works.
> 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
> but with support for explicit use of CompressionTools for decompression,
> where fields have been explicitly compressed with CompressionTools - the
> application will knows which need decompression)
> 5. Add a CompressionTools to my "test" index writer, generating explicitly
> compressed fields in the 2.9.1 "test" index
> 6. Test explicit decompression for relevant fields with CompressionTools in
> my 2.9.1 "test" index in my index readers
> 7. Upgrade my "test" index writer to 3.0.0
> 8. Have that "test" index writer push that 3.0.0 "test" index into
> production and see if the distributed 2.9.1 index readers can cope with it
> OK
> 9. Go through my index writers and index reader clients and systematically
> purge all of the Field.Store.COMPRESS fields and migrate to an explicit
> CompressionTools approach where applicable and no compression where
> applicable. During this phase I'll expect to have
> CompressionTools-compressed fields coexisting with their
> Field.Store.COMPRESS predecessors, where index reader client use of
> Field.Store.COMPRESS is in transit to the explicit decompression approach.
> 10. Upgrade my index writers to 3.0.0
> 11. Upgrade my index readers to 3.0.0
>
> I've simplified this a bit, because I shan't really be testing straight off
> in production(!) - I'll test the migration plan in a test cluster first;
> but
> this gives you the idea about the path.
>
> I wanted to know if I should expect problems with this plan. I'm depending
> on newer writers generating indexes for older readers and 3 is a major
> number upgrade. It looks like I can get away with this in version 3, but
> that's by no means a guarantee according to
> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>
> Does this sound like a good plan?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Weiwei Wang
Alex Wang
ēˇ‹å·¨å·¨
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Iā€™ve finished a upgrade from 2.4.1 to 3.0.0

What I do is like this:
1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
2. Use a 3.0.0 IndexReader to read the old version index and then use a
3.0.0 IndexWriter to write all the documents into a new index
3. Update QueryPaser to 3.0.0

I've redeployed my system and it works fine now.


On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:

> I have Lucene 2.3.1 code and indexes deployed in production in a
> distributed
> system and would like to bring everything up to date with 3.0.0 via 2.9.1.
>
> Here's my migration plan:
>
> 1. Add a index writer which generates a 2.9.1 "test" index
> 2. Have that "test" index writer push that 2.9.1 "test" index into
> production and see if the distributed 2.3.1 index readers can cope with it
> OK
> 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields) -
> we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if it
> works.
> 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
> but with support for explicit use of CompressionTools for decompression,
> where fields have been explicitly compressed with CompressionTools - the
> application will knows which need decompression)
> 5. Add a CompressionTools to my "test" index writer, generating explicitly
> compressed fields in the 2.9.1 "test" index
> 6. Test explicit decompression for relevant fields with CompressionTools in
> my 2.9.1 "test" index in my index readers
> 7. Upgrade my "test" index writer to 3.0.0
> 8. Have that "test" index writer push that 3.0.0 "test" index into
> production and see if the distributed 2.9.1 index readers can cope with it
> OK
> 9. Go through my index writers and index reader clients and systematically
> purge all of the Field.Store.COMPRESS fields and migrate to an explicit
> CompressionTools approach where applicable and no compression where
> applicable. During this phase I'll expect to have
> CompressionTools-compressed fields coexisting with their
> Field.Store.COMPRESS predecessors, where index reader client use of
> Field.Store.COMPRESS is in transit to the explicit decompression approach.
> 10. Upgrade my index writers to 3.0.0
> 11. Upgrade my index readers to 3.0.0
>
> I've simplified this a bit, because I shan't really be testing straight off
> in production(!) - I'll test the migration plan in a test cluster first;
> but
> this gives you the idea about the path.
>
> I wanted to know if I should expect problems with this plan. I'm depending
> on newer writers generating indexes for older readers and 3 is a major
> number upgrade. It looks like I can get away with this in version 3, but
> that's by no means a guarantee according to
> http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
>
> Does this sound like a good plan?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Weiwei Wang
Alex Wang
ēˇ‹å·¨å·¨
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

I have Lucene 2.3.1 code and indexes deployed in production in a distributed
system and would like to bring everything up to date with 3.0.0 via 2.9.1.

Here's my migration plan:

1. Add a index writer which generates a 2.9.1 "test" index 
2. Have that "test" index writer push that 2.9.1 "test" index into
production and see if the distributed 2.3.1 index readers can cope with it
OK
3. Upgrade my index writers to 2.9.1 (still using evil compressed fields) -
we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if it
works.
4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
but with support for explicit use of CompressionTools for decompression,
where fields have been explicitly compressed with CompressionTools - the
application will knows which need decompression)
5. Add a CompressionTools to my "test" index writer, generating explicitly
compressed fields in the 2.9.1 "test" index
6. Test explicit decompression for relevant fields with CompressionTools in
my 2.9.1 "test" index in my index readers
7. Upgrade my "test" index writer to 3.0.0 
8. Have that "test" index writer push that 3.0.0 "test" index into
production and see if the distributed 2.9.1 index readers can cope with it
OK
9. Go through my index writers and index reader clients and systematically
purge all of the Field.Store.COMPRESS fields and migrate to an explicit
CompressionTools approach where applicable and no compression where
applicable. During this phase I'll expect to have
CompressionTools-compressed fields coexisting with their
Field.Store.COMPRESS predecessors, where index reader client use of
Field.Store.COMPRESS is in transit to the explicit decompression approach.
10. Upgrade my index writers to 3.0.0 
11. Upgrade my index readers to 3.0.0

I've simplified this a bit, because I shan't really be testing straight off
in production(!) - I'll test the migration plan in a test cluster first; but
this gives you the idea about the path.

I wanted to know if I should expect problems with this plan. I'm depending
on newer writers generating indexes for older readers and 3 is a major
number upgrade. It looks like I can get away with this in version 3, but
that's by no means a guarantee according to
http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats

Does this sound like a good plan?


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

There was a thread a while back about how span queries don't enumerate
every possible span, but I can't remember if that included sometimes
missing payloads...

Mike

On Tue, Dec 8, 2009 at 7:34 PM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
> Howdy,
>
> I am wondering if anyone has seen
> NearSpansUnordered.getPayload() not return payloads that are
> verifiably accessible via IR.termPositions? It's a bit confusing
> because most of the time they're returned properly.
>
> I suspect the payload logic gets tripped up in
> NearSpansUnordered. I'll put together a test case, however the
> difficulty is that we're only seeing the issue with largish 800
> MB indexes, which could make the test case a little crazy.
>
> Jason
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

OK thanks for bringing closure!

Accidentally allowing 2 writers to write to the same index quickly
leads to corruption.  They are like the Betta fish: they fight to the
death, removing each others files, if you put them in the same cage.

Mike

On Wed, Dec 9, 2009 at 1:56 AM, Max Lynch <ihasmax@gmail.com> wrote:
> Hi Mike,
>
> Missed your response on this,
> What I was doing was physically removing index/write.lock if older than 8
> hours, allowing another process of my indexer to run.  I realize in
> hindsight that there is no reason why I should be doing this and it was
> really stupid.  I think I was under the impression one of my pylucene
> processes was hanging.
>
> On Fri, Oct 9, 2009 at 3:44 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> You can use o.a.l.index.CheckIndex to fix the index.  It will remove
>> references to any segments that are missing or have problems during
>> testing.  First run it without -fix to see what problems there are.
>> Then take a backup of the index.  Then run it with -fix.  The index
>> will lose all docs in those segments that it removes.
>>
>> Can you describe what led up to this?  Is it repeatable?
>>
>> Mike
>>
>> On Fri, Oct 9, 2009 at 12:37 AM, Max Lynch <ihasmax@gmail.com> wrote:
>> > Missed your response, thanks Bernd.
>> >
>> > I don't think that's it, since I haven't been executing any commands like
>> > that.  The only thing I could think of is corruption.  I've got the index
>> > backed up in case there is a way to fix it (it won't matter in a week or
>> so
>> > since I cull any documents older than 25 days).
>> >
>> > Is there a way to fix this?
>> >
>> > Thanks.
>> >
>> > On Thu, Oct 8, 2009 at 3:01 AM, Bernd Fondermann <
>> > bernd.fondermann@googlemail.com> wrote:
>> >
>> >> Hi Max
>> >>
>> >> just a guess: maybe you deleted all *.c source files in that area and
>> >> unintentionally deleted this index file, too.
>> >>
>> >>  Bernd
>> >>
>> >> On Fri, Oct 2, 2009 at 17:10, Max Lynch <ihasmax@gmail.com> wrote:
>> >> > I'm getting this error when I try to run my searcher and my indexer:
>> >> >
>> >> > Traceback (most recent call last):
>> >> > self.searcher = lucene.IndexSearcher(self.directory)
>> >> > JavaError: java.io.FileNotFoundException:
>> >> /home/spider/misc/index/_275c.cfs
>> >> > (No such file or directory)
>> >> >
>> >> > I don't know anything about the format of the Lucene index, but I
>> notice
>> >> I
>> >> > have several _275* files from b to j but no c.
>> >> >
>> >> > Any ideas?
>> >> >
>> >> > Thanks.
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hi all,
 
I have attached the code file and also the output screenshot. The problem is:
 
When i search for a query, it gives out the resultant documents containing the query but,
 
if two documents a and b contains that query,
Result is:
 
Link for a:
contents of a
contents of b
 
Link for b:
contents of a
contents of b
 
kindly go through the attached documents for the code, obtained output and the correct output
screenshots. 
Can anyone pls solve this problem.
I want the contents of the corresponding file alone to be displayed and the repetition should
be avoided.
 
Also how to rank my results / how to implement relevancy ranking.
Am a beginner and have no idea so kindly help me on this context.
 
Thanks,
Dhivya


      The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

Hi all,

The missing maven artifacts for the fast-vector-highlighter contrib of
Lucene Java in version 2.9.1 and 3.0.0 are now available at:

http://repo1.maven.org/maven2/org/apache/lucene/
http://repo2.maven.org/maven2/org/apache/lucene/

Uwe

-----
Uwe Schindler
uschindler@apache.org 
Apache Lucene Java Committer
Bremen, Germany
http://lucene.apache.org/java/docs/

> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Tuesday, December 08, 2009 10:41 PM
> To: java-dev@lucene.apache.org; general@lucene.apache.org
> Subject: RE: [VOTE] Push fast-vector-highlighter mvn artifacts for 3.0.0
> and 2.9.1
> 
> I got 3 binding votes from Grant, Mike, and Ted (and one from Simon, who
> was
> a big help on Sunday evening when I created the artifacts), so I push the
> maven artifacts onto the rsync repo in few minutes.
> 
> Thanks!
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Tuesday, December 08, 2009 7:03 PM
> > To: java-dev@lucene.apache.org
> > Subject: [VOTE] Push fast-vector-highlighter mvn artifacts for 3.0.0 and
> > 2.9.1
> >
> > Sorry,
> >
> > I initially didn't want to start a vote, as Grant only proposed to
> "maybe
> > start one". But nobody responded (esp. to the questions in this mail) I
> > ask
> > again, an I will start the vote for now.
> >
> >
> ==========================================================================
> > ==
> > Please vote, that the missing artifacts for of fast-verctor-highlighter
> of
> > Lucene Java 2.9.1 and 3.0.0 should be pushed to repoX.maven.org.
> >
> > You can find the artifacts here:
> > http://people.apache.org/~uschindler/staging-area/
> >
> > This dir contains only the maven folder to be copied to maven-rsync dir
> on
> > p.a.o. The top-level version in the maven metadata is 3.0.0, which
> > conforms
> > to the current state on maven (so during merging both folders during
> > build,
> > I set preference to metadata.xml of 3.0.0).
> >
> > All files are signed by my PGP key (even the 2.9.1 ones; that release
> was
> > originally built by Mike McCandless).
> >
> ==========================================================================
> > ==
> >
> > What I additionally found out until now (because Simon nagged me):
> >
> > If you compare the JAR files inside the binary ZIP file from the apache
> > archive and the JAR files directly published on maven (for the other
> > contribs), the MD5s/SHA1s are different even as they are created from
> the
> > same source code (because the timestamps inside the JAR are different,
> for
> > 2.9.1 another JDK compiler/platform was used). This interestingly does
> not
> > apply to lucene-core.jar in 3.0. Because of that I see no problem with
> > this
> > maven release, even that they are not the orginal JAR files from the
> > binary
> > distrib.
> >
> > What is not nice, is that the svn revision number in the manifest is
> > different, but else is exactly the same, see my comments below in
> earlier
> > mails about changing the ant script for showing the SVN rev of the last
> > changed file.
> >
> > So if nobody objects to release these rebuild jar files, all signed by
> my
> > key, I would like to simply put them on the maven-rsync folder.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> > > Sent: Tuesday, December 08, 2009 6:48 PM
> > > To: java-dev@lucene.apache.org
> > > Subject: Re: (NAG) Push fast-vector-highlighter mvn artifacts for 3.0
> > and
> > > 2.9
> > >
> > >
> > > : What to do now, any votes on adding the missing maven artifacts for
> > > : fast-vector-highlighter to 2.9.1 and 3.0.0 on the apache maven
> > > reposititory?
> > >
> > > It's not even clear to me that anything special needs to be done
> before
> > > publishing those jars to maven.  2.9.1 and 3.0.0 were already voted on
> > and
> > > released -- including all of the source code in them.
> > >
> > > The safest bet least likely to anger the process gods is just to call
> a
> > > vote (new thread with VOTE in the subject) and cast a vote ...
> > considering
> > > the sources has already been reviewed it should go pretty quick.
> > >
> > > :
> > > : > I rebuilt the maven-dir for 2.9.1 and 3.0.0, merged them (3.0.0 is
> > > top-
> > > : > level
> > > : > version) and extracted only fast-vector-highlighter:
> > > : >
> > > : > http://people.apache.org/~uschindler/staging-area/
> > > : >
> > > : > I will copy this dir to the maven folder on people.a.o, when I got
> > > votes
> > > : > (how many)? At least someone should check the signatures.
> > > : >
> > > : > By the way, we have a small error in our ant build.xml that
> inserts
> > > : > svnversion into the manifest file. This version is not the version
> > of
> > > the
> > > : > last changed item (would be svnversion -c) but the current svn
> > > version,
> > > : > even
> > > : > that I checked out the corresponding tags. It's no problem at all,
> > but
> > > not
> > > : > very nice.
> > > : >
> > > : > Maybe we should change build.xml to call "svnversion -c" in
> future,
> > to
> > > get
> > > : > the real number.
> > > : >
> > > : > Uwe
> > > : >
> > > : > -----
> > > : > Uwe Schindler
> > > : > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > : > http://www.thetaphi.de
> > > : > eMail: uwe@thetaphi.de
> > > : >
> > > : >
> > > : > > -----Original Message-----
> > > : > > From: Grant Ingersoll [mailto:gsingers@apache.org]
> > > : > > Sent: Saturday, December 05, 2009 10:26 PM
> > > : > > To: java-dev@lucene.apache.org
> > > : > > Subject: Re: Push fast-vector-highlighter mvn artifacts for 3.0
> > and
> > > 2.9
> > > : > >
> > > : > > I suppose we could put up the artifacts on a dev site and then
> we
> > > could
> > > : > > vote to release both of them pretty quickly.  I think that
> should
> > be
> > > : > easy
> > > : > > to do, since it pretty much only involves verifying the jar and
> > the
> > > : > > signatures.
> > > : > >
> > > : > > On Dec 5, 2009, at 1:03 PM, Simon Willnauer wrote:
> > > : > >
> > > : > > > hi folks,
> > > : > > > The maven artifacts for fast-vector-highlighter have never
> been
> > > pushed
> > > : > > > since it was released because there were no pom.xml.template
> > > inside
> > > : > > > the module. I added a pom file a day ago in the context of
> > > : > > > LUCENE-2107. I already talked to uwe and grant how to deal
> with
> > > this
> > > : > > > issues and if we should push the artifact for Lucene 2.9 /
> 3.0.
> > > Since
> > > : > > > this is only a metadata file we could consider rebuilding
the
> > > : > > > artefacts and publish them for those releases. I can not
> > remember
> > > that
> > > : > > > anything like that happened before, so we should discuss how
> to
> > > deal
> > > : > > > with this situation and if we should wait until 3.1.
> > > : > > >
> > > : > > > simon
> > > : > > >
> > > : > > > --------------------------------------------------------------
> --
> > --
> > > ---
> > > : > > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > : > > > For additional commands, e-mail: java-dev-
> help@lucene.apache.org
> > > : > > >
> > > : > >
> > > : > >
> > > : > >
> > > : > > ----------------------------------------------------------------
> --
> > --
> > > -
> > > : > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > : > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > > : >
> > > : >
> > > : >
> > > : > ------------------------------------------------------------------
> --
> > -
> > > : > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > : > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > > :
> > > :
> > > :
> > > : --------------------------------------------------------------------
> -
> > > : To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > : For additional commands, e-mail: java-dev-help@lucene.apache.org
> > > :
> > >
> > >
> > >
> > > -Hoss
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Thanks, Uwe. I've found the problem. the updateTime field is lost when i
converted my index from an older version.

Another question, is there any detailed tutorial about Lucene 3.0.0?

2009/12/9 Uwe Schindler <uwe@thetaphi.de>

> How did you index your date?
>
> I would suggest to reindex the date using NumericField! And then query
> using
> NumericRangeQuery. If reindexing is not possible the Query like you have
> done, should work. Please give us examples of how you indexed and how you
> query.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
> > Sent: Wednesday, December 09, 2009 4:23 AM
> > To: java-user@lucene.apache.org
> > Subject: HOW to do date range searchi in 3.0
> >
> > Hi, all
> >      I need to do a date range search like date:[a previous time to null]
> > I used a filter to do this job, the code  is shown below:
> >     Calendar c = Calendar.getInstance();
> >     c.setTimeInMillis(c.getTimeInMillis() -
> > parameter.getRecentUpdateConstraint()
> >         * RosaCrawlerConstants.ONE_DAY_IN_MILLISECOND);
> >     String fromTime = DateTools.dateToString(c.getTime(),
> > DateTools.Resolution.DAY);
> >     Query updateTimeRange = new
> > TermRangeQuery("updateTime",fromTime,null,true,false);
> >     query.add(updateTimeRange, BooleanClause.Occur.MUST);
> >
> > However, it doesn't work as before in version 2.4.1(I'm updating my
> > project
> > from version 2.4.1 to lucenen 3.0.0)
> >
> > Could anybody here offer me a solution?
> > --
> > Weiwei Wang
> > Alex Wang
> > ēˇ‹å·¨å·¨
> > Room 403, Mengmin Wei Building
> > Computer Science Department
> > Gulou Campus of Nanjing University
> > Nanjing, P.R.China, 210093
> >
> > Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Weiwei Wang
Alex Wang
ēˇ‹å·¨å·¨
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

How did you index your date?

I would suggest to reindex the date using NumericField! And then query using
NumericRangeQuery. If reindexing is not possible the Query like you have
done, should work. Please give us examples of how you indexed and how you
query.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
> Sent: Wednesday, December 09, 2009 4:23 AM
> To: java-user@lucene.apache.org
> Subject: HOW to do date range searchi in 3.0
> 
> Hi, all
>      I need to do a date range search like date:[a previous time to null]
> I used a filter to do this job, the code  is shown below:
>     Calendar c = Calendar.getInstance();
>     c.setTimeInMillis(c.getTimeInMillis() -
> parameter.getRecentUpdateConstraint()
>         * RosaCrawlerConstants.ONE_DAY_IN_MILLISECOND);
>     String fromTime = DateTools.dateToString(c.getTime(),
> DateTools.Resolution.DAY);
>     Query updateTimeRange = new
> TermRangeQuery("updateTime",fromTime,null,true,false);
>     query.add(updateTimeRange, BooleanClause.Occur.MUST);
> 
> However, it doesn't work as before in version 2.4.1(I'm updating my
> project
> from version 2.4.1 to lucenen 3.0.0)
> 
> Could anybody here offer me a solution?
> --
> Weiwei Wang
> Alex Wang
> $B2&V[V[(B
> Room 403, Mengmin Wei Building
> Computer Science Department
> Gulou Campus of Nanjing University
> Nanjing, P.R.China, 210093
> 
> Homepage: http://cs.nju.edu.cn/rl/weiweiwang


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hi Mike,

Missed your response on this,
What I was doing was physically removing index/write.lock if older than 8
hours, allowing another process of my indexer to run.  I realize in
hindsight that there is no reason why I should be doing this and it was
really stupid.  I think I was under the impression one of my pylucene
processes was hanging.

On Fri, Oct 9, 2009 at 3:44 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> You can use o.a.l.index.CheckIndex to fix the index.  It will remove
> references to any segments that are missing or have problems during
> testing.  First run it without -fix to see what problems there are.
> Then take a backup of the index.  Then run it with -fix.  The index
> will lose all docs in those segments that it removes.
>
> Can you describe what led up to this?  Is it repeatable?
>
> Mike
>
> On Fri, Oct 9, 2009 at 12:37 AM, Max Lynch <ihasmax@gmail.com> wrote:
> > Missed your response, thanks Bernd.
> >
> > I don't think that's it, since I haven't been executing any commands like
> > that.  The only thing I could think of is corruption.  I've got the index
> > backed up in case there is a way to fix it (it won't matter in a week or
> so
> > since I cull any documents older than 25 days).
> >
> > Is there a way to fix this?
> >
> > Thanks.
> >
> > On Thu, Oct 8, 2009 at 3:01 AM, Bernd Fondermann <
> > bernd.fondermann@googlemail.com> wrote:
> >
> >> Hi Max
> >>
> >> just a guess: maybe you deleted all *.c source files in that area and
> >> unintentionally deleted this index file, too.
> >>
> >>  Bernd
> >>
> >> On Fri, Oct 2, 2009 at 17:10, Max Lynch <ihasmax@gmail.com> wrote:
> >> > I'm getting this error when I try to run my searcher and my indexer:
> >> >
> >> > Traceback (most recent call last):
> >> > self.searcher = lucene.IndexSearcher(self.directory)
> >> > JavaError: java.io.FileNotFoundException:
> >> /home/spider/misc/index/_275c.cfs
> >> > (No such file or directory)
> >> >
> >> > I don't know anything about the format of the Lucene index, but I
> notice
> >> I
> >> > have several _275* files from b to j but no c.
> >> >
> >> > Any ideas?
> >> >
> >> > Thanks.
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Hi, all
     I need to do a date range search like date:[a previous time to null]
I used a filter to do this job, the code  is shown below:
    Calendar c = Calendar.getInstance();
    c.setTimeInMillis(c.getTimeInMillis() -
parameter.getRecentUpdateConstraint()
        * RosaCrawlerConstants.ONE_DAY_IN_MILLISECOND);
    String fromTime = DateTools.dateToString(c.getTime(),
DateTools.Resolution.DAY);
    Query updateTimeRange = new
TermRangeQuery("updateTime",fromTime,null,true,false);
    query.add(updateTimeRange, BooleanClause.Occur.MUST);

However, it doesn't work as before in version 2.4.1(I'm updating my project
from version 2.4.1 to lucenen 3.0.0)

Could anybody here offer me a solution?
-- 
Weiwei Wang
Alex Wang
ēˇ‹å·¨å·¨
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

Howdy,

I am wondering if anyone has seen
NearSpansUnordered.getPayload() not return payloads that are
verifiably accessible via IR.termPositions? It's a bit confusing
because most of the time they're returned properly.

I suspect the payload logic gets tripped up in
NearSpansUnordered. I'll put together a test case, however the
difficulty is that we're only seeing the issue with largish 800
MB indexes, which could make the test case a little crazy.

Jason

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Sorry wrong word, Germans often have the problem with English "must". It has
to be to be "but you must not".

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Steven A Rowe [mailto:sarowe@syr.edu]
> Sent: Tuesday, December 08, 2009 8:42 PM
> To: java-user@lucene.apache.org
> Subject: RE: TopFieldDocCollector and v3.0.0
> 
> Hi Uwe,
> 
> On 12/08/2009 at 9:40 AM, Uwe Schindler wrote:
> > After the move to 3.0, you can (but you must not) further update
> > your code to use generics, which is not really needed but will
> > remove all compiler warnings.
> 
> This sounds like you're telling people that although they are able to
> update their code to use generics, it is forbidden.
> 
> I'm sure, though, that you mean that they are not required to do so:
> something like "but you need not" rather than "but you must not".
> 
> Steve
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hello Tom and Erick,
                          I am really sorry for posting such a dull
question. Meanwhile I have explored a few other parts of API,
fortunately I have found a place which could exaclty fit for my case.
Thanks for patiently trying to understand my question.. and warning
me.
Bye.

On Tue, Dec 8, 2009 at 9:21 PM, Tom Hill <solr-list@worldware.com> wrote:
> If you tell us WHY you want to do this, rather than HOW you want to do it,
> the chances are much better that someone can help.
>
> What's the business motivation here?  What does the end user want to
> achieve?
>
> Tom
>
> On Tue, Dec 8, 2009 at 8:16 AM, Phanindra Reva <reva.phanindra@gmail.com>wrote:
>
>> Hello,
>>        Thanks for the reply. *strange* was expected. I am trying to
>> store field names as payloads, so I need unedited field names during
>> analysis part. And later my plan is to replace all the field names
>> with a default value and then store the document in the index. So, If
>> its possible to get the reference of the Document after the analysis,
>> I could modify all the field names. Even a way of modifying ( of
>> course , it should be after analysis and before adding to the index )
>> the field-name values that are going to be added to the index will
>> suffice.
>>     I guess.. this time you feel its much more strange, but that's my
>> task for which above mentioned is one way to accomplish.
>> Thanks.
>>
>> On Tue, Dec 8, 2009 at 4:55 PM, Erick Erickson <erickerickson@gmail.com>
>> wrote:
>> > You're right, it *does* seem strange <G>....
>> >
>> > I'm having a really hard time imagining a use-case
>> > for this capability, so it's hard to suggest
>> > an approach. Perhaps you could supply
>> > an outline of your use-case? This may be
>> > an XY problem.
>> >
>> > Best
>> > Erick
>> >
>> > On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <
>> reva.phanindra@gmail.com>wrote:
>> >
>> >> Hello All,
>> >>              I am a newbie using Lucene. To be brief, I am just
>> >> wondering whether is there a point where we get the access to the
>> >> org.apache.lucene.document.Document (which is being indexed at the
>> >> moment)  after the analysing part is completed but exactly before it
>> >> is added to the index. My whole aim is to modify all the field names
>> >> present in the document before its being added to the index, but I
>> >> need those field names un-edited during the analysis part.
>> >>   If it seems strange.. please don't mind.
>> >> Thanks.
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

If you tell us WHY you want to do this, rather than HOW you want to do it,
the chances are much better that someone can help.

What's the business motivation here?  What does the end user want to
achieve?

Tom

On Tue, Dec 8, 2009 at 8:16 AM, Phanindra Reva <reva.phanindra@gmail.com>wrote:

> Hello,
>        Thanks for the reply. *strange* was expected. I am trying to
> store field names as payloads, so I need unedited field names during
> analysis part. And later my plan is to replace all the field names
> with a default value and then store the document in the index. So, If
> its possible to get the reference of the Document after the analysis,
> I could modify all the field names. Even a way of modifying ( of
> course , it should be after analysis and before adding to the index )
> the field-name values that are going to be added to the index will
> suffice.
>     I guess.. this time you feel its much more strange, but that's my
> task for which above mentioned is one way to accomplish.
> Thanks.
>
> On Tue, Dec 8, 2009 at 4:55 PM, Erick Erickson <erickerickson@gmail.com>
> wrote:
> > You're right, it *does* seem strange <G>....
> >
> > I'm having a really hard time imagining a use-case
> > for this capability, so it's hard to suggest
> > an approach. Perhaps you could supply
> > an outline of your use-case? This may be
> > an XY problem.
> >
> > Best
> > Erick
> >
> > On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <
> reva.phanindra@gmail.com>wrote:
> >
> >> Hello All,
> >>              I am a newbie using Lucene. To be brief, I am just
> >> wondering whether is there a point where we get the access to the
> >> org.apache.lucene.document.Document (which is being indexed at the
> >> moment)  after the analysing part is completed but exactly before it
> >> is added to the index. My whole aim is to modify all the field names
> >> present in the document before its being added to the index, but I
> >> need those field names un-edited during the analysis part.
> >>   If it seems strange.. please don't mind.
> >> Thanks.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Hi Uwe,

On 12/08/2009 at 9:40 AM, Uwe Schindler wrote:
> After the move to 3.0, you can (but you must not) further update
> your code to use generics, which is not really needed but will
> remove all compiler warnings.

This sounds like you're telling people that although they are able to update their code to
use generics, it is forbidden.

I'm sure, though, that you mean that they are not required to do so: something like "but you
need not" rather than "but you must not".

Steve

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

If you store the field unanalyzed it will be indexed as is.  You can
then search for it via a TermQuery, or use QueryParser with
PerFieldAnalyzerWrapper specifying KeywordAnalyzer for the field
containing this character.

Another approach is to replace the % with something easier to work
with.  You could do this yourself or, I think, with MappingCharFilter.

Personally I'd probably replace "%" with "percent" somewhere in my
code with a simple string replacement.


--
Ian.



On Tue, Dec 8, 2009 at 5:02 PM, kanayo <richies4all@gmail.com> wrote:
>
> Thanks for your reply Erick.
>
> In Luke, its also not working. I tried to retrieving values from the field
> by specifying the field as the search field and then specify % as the search
> parameter which using StandardAnalyzer but nothing is displayed. Also while
> Luke shows the query details for other search values, it dosent show query
> details for search value of %.
>
> I think it is not Tokenized in the index. Is there anything else i can do to
> be able to retrieve values from fields comprising of just %?
>
> Thanks for your assistance.
>
>
> Erick Erickson wrote:
>>
>> Try printing out query.toString() to see what's actually being
>> sent to the searcher.
>>
>> You can try the same thing in Luke, specifying StandardAnalyzer
>> to parse queries.
>>
>> Are you sure you're specifying the fields in the query and not just the
>> '%'? That would go against your default field.
>>
>> When you say that you can see the fields in luke, are you storing the
>> field?
>> Because what you may be seeing is the *stored* value rather than the
>> *tokens*.
>> Make sure you're looking at the tokens in Luke..
>>
>> If none of that helps, could you post a code snippet or two (index and
>> query)?
>>
>> Best
>> Erick
>>
>> On Tue, Dec 8, 2009 at 11:04 AM, kanayo <richies4all@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I am a newbie to lucene. I am using Standard Analyzer in my lucene
>>> project.
>>> I am indexing some fields which may contain only "%" as a field value and
>>> it
>>> indexes fine and i can view the value against the field in the index
>>> using
>>> Luke.
>>>
>>> However when i try to retrieve the same field using indexsearcher and
>>> passing "%" as a query parameter nothing is retrieved. It is simply being
>>> ignored. I have also tried to escape the "%" while searching but still no
>>> results.
>>>
>>> Is there anything am not doing right?
>>>
>>> Thanks in advance for your assistance.
>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696184.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696993.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Thanks for your reply Erick.

In Luke, its also not working. I tried to retrieving values from the field
by specifying the field as the search field and then specify % as the search
parameter which using StandardAnalyzer but nothing is displayed. Also while
Luke shows the query details for other search values, it dosent show query
details for search value of %.

I think it is not Tokenized in the index. Is there anything else i can do to
be able to retrieve values from fields comprising of just %?

Thanks for your assistance.


Erick Erickson wrote:
> 
> Try printing out query.toString() to see what's actually being
> sent to the searcher.
> 
> You can try the same thing in Luke, specifying StandardAnalyzer
> to parse queries.
> 
> Are you sure you're specifying the fields in the query and not just the
> '%'? That would go against your default field.
> 
> When you say that you can see the fields in luke, are you storing the
> field?
> Because what you may be seeing is the *stored* value rather than the
> *tokens*.
> Make sure you're looking at the tokens in Luke..
> 
> If none of that helps, could you post a code snippet or two (index and
> query)?
> 
> Best
> Erick
> 
> On Tue, Dec 8, 2009 at 11:04 AM, kanayo <richies4all@gmail.com> wrote:
> 
>>
>> Hi,
>>
>> I am a newbie to lucene. I am using Standard Analyzer in my lucene
>> project.
>> I am indexing some fields which may contain only "%" as a field value and
>> it
>> indexes fine and i can view the value against the field in the index
>> using
>> Luke.
>>
>> However when i try to retrieve the same field using indexsearcher and
>> passing "%" as a query parameter nothing is retrieved. It is simply being
>> ignored. I have also tried to escape the "%" while searching but still no
>> results.
>>
>> Is there anything am not doing right?
>>
>> Thanks in advance for your assistance.
>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696184.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> 

-- 
View this message in context: http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696993.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Try printing out query.toString() to see what's actually being
sent to the searcher.

You can try the same thing in Luke, specifying StandardAnalyzer
to parse queries.

Are you sure you're specifying the fields in the query and not just the
'%'? That would go against your default field.

When you say that you can see the fields in luke, are you storing the field?
Because what you may be seeing is the *stored* value rather than the
*tokens*.
Make sure you're looking at the tokens in Luke..

If none of that helps, could you post a code snippet or two (index and
query)?

Best
Erick

On Tue, Dec 8, 2009 at 11:04 AM, kanayo <richies4all@gmail.com> wrote:

>
> Hi,
>
> I am a newbie to lucene. I am using Standard Analyzer in my lucene project.
> I am indexing some fields which may contain only "%" as a field value and
> it
> indexes fine and i can view the value against the field in the index using
> Luke.
>
> However when i try to retrieve the same field using indexsearcher and
> passing "%" as a query parameter nothing is retrieved. It is simply being
> ignored. I have also tried to escape the "%" while searching but still no
> results.
>
> Is there anything am not doing right?
>
> Thanks in advance for your assistance.
>
>
> --
> View this message in context:
> http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696184.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Hello,
        Thanks for the reply. *strange* was expected. I am trying to
store field names as payloads, so I need unedited field names during
analysis part. And later my plan is to replace all the field names
with a default value and then store the document in the index. So, If
its possible to get the reference of the Document after the analysis,
I could modify all the field names. Even a way of modifying ( of
course , it should be after analysis and before adding to the index )
the field-name values that are going to be added to the index will
suffice.
     I guess.. this time you feel its much more strange, but that's my
task for which above mentioned is one way to accomplish.
Thanks.

On Tue, Dec 8, 2009 at 4:55 PM, Erick Erickson <erickerickson@gmail.com> wrote:
> You're right, it *does* seem strange <G>....
>
> I'm having a really hard time imagining a use-case
> for this capability, so it's hard to suggest
> an approach. Perhaps you could supply
> an outline of your use-case? This may be
> an XY problem.
>
> Best
> Erick
>
> On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <reva.phanindra@gmail.com>wrote:
>
>> Hello All,
>>              I am a newbie using Lucene. To be brief, I am just
>> wondering whether is there a point where we get the access to the
>> org.apache.lucene.document.Document (which is being indexed at the
>> moment)  after the analysing part is completed but exactly before it
>> is added to the index. My whole aim is to modify all the field names
>> present in the document before its being added to the index, but I
>> need those field names un-edited during the analysis part.
>>   If it seems strange.. please don't mind.
>> Thanks.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Hi,

I am a newbie to lucene. I am using Standard Analyzer in my lucene project. 
I am indexing some fields which may contain only "%" as a field value and it
indexes fine and i can view the value against the field in the index using
Luke.

However when i try to retrieve the same field using indexsearcher and
passing "%" as a query parameter nothing is retrieved. It is simply being
ignored. I have also tried to escape the "%" while searching but still no
results.

Is there anything am not doing right?

Thanks in advance for your assistance.


-- 
View this message in context: http://old.nabble.com/Problem-searching-field-with---as-value-tp26696184p26696184.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

You're right, it *does* seem strange <G>....

I'm having a really hard time imagining a use-case
for this capability, so it's hard to suggest
an approach. Perhaps you could supply
an outline of your use-case? This may be
an XY problem.

Best
Erick

On Tue, Dec 8, 2009 at 10:12 AM, Phanindra Reva <reva.phanindra@gmail.com>wrote:

> Hello All,
>              I am a newbie using Lucene. To be brief, I am just
> wondering whether is there a point where we get the access to the
> org.apache.lucene.document.Document (which is being indexed at the
> moment)  after the analysing part is completed but exactly before it
> is added to the index. My whole aim is to modify all the field names
> present in the document before its being added to the index, but I
> need those field names un-edited during the analysis part.
>   If it seems strange.. please don't mind.
> Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>