Mailing-List: contact user-help@lucy.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@lucy.apache.org
To: user@lucy.apache.org
References: <CY1PR06MB2172D9FFA47BAFD82A51351794820@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB21724B8E85F44099F82B64CF94850@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB21724E5BD5F52E852B3CFBBE94850@CY1PR06MB2172.namprd06.prod.outlook.com>
 <BY2PR06MB2167AC743BB46B64BBFACFAA94840@BY2PR06MB2167.namprd06.prod.outlook.com>
 <CY1PR06MB217243235E91EA8613C7297694870@CY1PR06MB2172.namprd06.prod.outlook.com>
 <415e1dbe-27fe-3dab-2ed4-8ca57ebfab99@aevum.de>
 <CY1PR06MB217246E30E66419221C9F3BB94870@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB217219578F97831B8D916B6F94860@CY1PR06MB2172.namprd06.prod.outlook.com>
 <e0816e0a-f4a1-b5bb-2b68-4de7827161d8@aevum.de>
 <CY1PR06MB2172BC1929F00359649B9D1394860@CY1PR06MB2172.namprd06.prod.outlook.com>
 <61261e18-7351-17fd-00df-c26f564f7d9f@aevum.de>
 <CY1PR06MB21722379929C7C842147679894980@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB2172B3B1E371164ED8D9E682949B0@CY1PR06MB2172.namprd06.prod.outlook.com>
From: Nick Wellnhofer <wellnhofer@aevum.de>
Message-ID: <fd7c6553-1fbc-66ea-a1c5-82ddde79535c@aevum.de>
Date: Fri, 16 Dec 2016 22:22:09 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.5.1
MIME-Version: 1.0
In-Reply-To: <CY1PR06MB2172B3B1E371164ED8D9E682949B0@CY1PR06MB2172.namprd06.prod.outlook.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c
 line 119
archived-at: Fri, 16 Dec 2016 21:22:17 -0000

On 13/12/2016 18:05, Gupta, Rajiv wrote:
> After I create directory by myself I'm getting this error:

Which directory do you try to create? I wouldn't try to make manual changes 
inside Lucy's index directory. This will only make things worse.

> Can't open '/u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1/seg_fd/lexicon-7.ixix': Invalid argument
> 20161211 182109 [] *    LUCY_FSFolder_Local_Open_FileHandle_IMP at core/Lucy/Store/FSFolder.c line 118
> 20161211 182109 [] *    LUCY_Folder_Local_Open_In_IMP at core/Lucy/Store/Folder.c line 101
> 20161211 182109 [] *    LUCY_Folder_Open_In_IMP at core/Lucy/Store/Folder.c line 75
>
> There are two more failures they also failed due so similar reasons
>
> rename from '/u/smoke/presub/logs/cit-fg-adr-ndo-rtp.rajivg.1481473039.49384_cmode_1of1/.lucyindex/1/schema.temp' to '/u/smoke/presub/logs/cit-fg-adr-ndo-rtp.rajivg.1481473039.49384_cmode_1of1/.lucyindex/1/schema_e4.json' failed: No such file or directory
>
> Can't delete 'lexicon-3.ix'
>
> I believe all three are related to race condition while doing parallel indexing and should go away with retries. However my retries started failing with different error which is strange to me as if directory already exists shouldn't it skip from create attempt.
>
> 20161211 182109 [] * FAIL: [FAILED]: Retrying to add doc at path: /u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1 :  Couldn't create directory '/u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1': No such file or directory
> 20161211 182109 [] *    LUCY_FSFolder_Initialize_IMP at core/Lucy/Store/FSFolder.c line 102
>
> So my all retry attempts were also failed.

These errors still look like multiple processes are modifying the index at the 
same time. Are you really sure that every indexer is created with an 
IndexManager and that every IndexManager is created with a `host` argument 
that is unique to each machine?

All these errors mean that there's something fundamentally wrong with your 
code or that you hit a bug in Lucy. The only type of error where it makes 
sense to retry is LockErr. All other errors are mostly fatal and could result 
in index corruption. Retrying will only mask an underlying problem in this case.

Unfortunately, I'm unable to help unless you provide some kind of 
self-contained, reproducible test case. I'm aware that this isn't easy, 
especially with multiple clients writing to a shared volume.

As I already hinted at, you might want to reconsider your architecture and use 
some kind of search server that uses an index on a local filesystem. There are 
ready-made platforms on top of Lucy like Dezi, but it isn't too hard to roll 
your own solution. This should result in better performance and makes behavior 
of your code more predictable.

Nick