Mailing-List: contact user-help@lucy.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@lucy.apache.org
Sender: Peter Karman <peknet@gmail.com>
Reply-To: peter@peknet.com
References: <CY1PR06MB2172D9FFA47BAFD82A51351794820@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB21725B216E89D7369DA6F0AA94820@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB2172F53E8B1153BEE8B50DC894820@CY1PR06MB2172.namprd06.prod.outlook.com>
To: user@lucy.apache.org
From: Peter Karman <peter@peknet.com>
Message-ID: <667883c5-dfa9-e2f9-fb9f-cf9475d41f30@peknet.com>
Date: Tue, 6 Dec 2016 16:22:10 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
 Gecko/20100101 Thunderbird/45.5.1
MIME-Version: 1.0
In-Reply-To: <CY1PR06MB2172F53E8B1153BEE8B50DC894820@CY1PR06MB2172.namprd06.prod.outlook.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [lucy-user] RE: LUCY_Folder_Open_Out_IMP at
 core/Lucy/Store/Folder.c line 119
archived-at: Tue, 06 Dec 2016 22:22:18 -0000

Gupta, Rajiv wrote on 12/6/16 3:16 PM:
> I thought since I'm doing read and write together I may be getting file error
> so I tried to use FastUpdate method described here -
> http://lucy.apache.org/docs/perl/Lucy/Docs/Cookbook/FastUpdates.html
>
> But now I'm more frequently getting below error. Error input 57 too high
> 20161206 150630 []  S_fibonacci at core/Lucy/Index/IndexManager.c line 129
>
> My use case is. While my application generating multiple logs, I'm indexing
> them parallelly. To achieve this I'm storing docs at multiple locations at
> each directory level. In a directory there could be multiple log files so for
> that directory I'm having one indexer directory. When file get closed I
> insert an end marker doc to indicate that indexing on that file is done.
> However, sometimes same file get open multiple times with additional data. In
> such case I search in existing indexing directory if there is any end marker
> is set, if there is end marker then I delete that end marker, and index
> additional data and again insert end marker. In this process I search as well
> as write at the same time.
>
> I started seeing these type of error only after I inserted the logic of
> search and deleting. Should I catch this and retry? How many docs I should
> limit to commit together?


It wasn't clear to me from your description, but you should be running only one 
indexer at time per invindex.

You should also destroy any open searchers once the invindex changes (the 
indexer commits).

You can see here how Dezi approaches this:

https://metacpan.org/source/KARMAN/Dezi-App-0.014/lib/Dezi/Lucy/Searcher.pm#L406

TL;DR Dezi keeps track of a invindex header file with a UUID in it, which 
changes whenever the indexer finishes. Both the UUID and a md5 checksum of the 
header file are checked on every search, and the searcher is destroyed and a new 
one created if the old searcher is stale.


-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com