lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <mar...@rectangular.com>
Subject Re: [lucy-user] Lucy unable to find an index file
Date Fri, 09 Aug 2013 15:48:27 GMT
On Fri, Aug 9, 2013 at 5:29 AM, Edwin Crockford <ecrockford@invicro.com> wrote:
> I believe we
> have traced the issue back to an interaction between two different systems
> (one doing bulk updates and another doing on the fly single document
> indexing) attempting updates at the same time. I think there was a way
> around the locking that caused the problem, does that seem plausable?

Yes, that makes sense.  The error can be explained by having two Indexers
trying to write to the same segment.  One of them happens to delete the temp
file "lextemp" first, and then the other can't find it and throws an
exception.

Only one Indexer may operate on a given index at a time.  A BackgroundMerger
may operate at the same time as an Indexer, but even it must acquire the write
lock briefly (once at the start of its run and once again at the end).  While
Lucy's locking APIs provide the technical capacity to disable the locking
mechanism, it is not generally possible to get around the need for locking in
order to coordinate write access to an index.

> The error was reproducible to some extent but went away when we shut down
> one of the systems.
>
> File system is standard UNIX, Centos 6
>
> Reindexing refers to the bulk indexer (mentioned above) that rebuilds the
> index from scratch.
>
> No permissions issue that are obvious, both indexers were running as the
> same user.
>
> The docs are added in blocks of around 10 by the bulk indexer, we always
> populate all the fields
>
> Any ideas would be helpful

If the "bulk indexer" does not change any content, you might look into
BackgroundMerger.

If it changes content, then perhaps consider building the replacement index
off to the side and queueing new docs, then adding the new docs after the
bulk update finishes and finally moving the index into place.

Marvin Humphrey

Mime
View raw message