lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Karman <pe...@peknet.com>
Subject Re: [lucy-user] RE: LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c line 119
Date Tue, 06 Dec 2016 22:22:10 GMT
Gupta, Rajiv wrote on 12/6/16 3:16 PM:
> I thought since I'm doing read and write together I may be getting file error
> so I tried to use FastUpdate method described here -
> http://lucy.apache.org/docs/perl/Lucy/Docs/Cookbook/FastUpdates.html
>
> But now I'm more frequently getting below error. Error input 57 too high
> 20161206 150630 []  S_fibonacci at core/Lucy/Index/IndexManager.c line 129
>
> My use case is. While my application generating multiple logs, I'm indexing
> them parallelly. To achieve this I'm storing docs at multiple locations at
> each directory level. In a directory there could be multiple log files so for
> that directory I'm having one indexer directory. When file get closed I
> insert an end marker doc to indicate that indexing on that file is done.
> However, sometimes same file get open multiple times with additional data. In
> such case I search in existing indexing directory if there is any end marker
> is set, if there is end marker then I delete that end marker, and index
> additional data and again insert end marker. In this process I search as well
> as write at the same time.
>
> I started seeing these type of error only after I inserted the logic of
> search and deleting. Should I catch this and retry? How many docs I should
> limit to commit together?


It wasn't clear to me from your description, but you should be running only one 
indexer at time per invindex.

You should also destroy any open searchers once the invindex changes (the 
indexer commits).

You can see here how Dezi approaches this:

https://metacpan.org/source/KARMAN/Dezi-App-0.014/lib/Dezi/Lucy/Searcher.pm#L406

TL;DR Dezi keeps track of a invindex header file with a UUID in it, which 
changes whenever the indexer finishes. Both the UUID and a md5 checksum of the 
header file are checked on every search, and the searcher is destroyed and a new 
one created if the old searcher is stale.


-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Mime
View raw message