lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Wellnhofer <wellnho...@aevum.de>
Subject Re: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c line 119
Date Fri, 16 Dec 2016 21:22:09 GMT
On 13/12/2016 18:05, Gupta, Rajiv wrote:
> After I create directory by myself I'm getting this error:

Which directory do you try to create? I wouldn't try to make manual changes 
inside Lucy's index directory. This will only make things worse.

> Can't open '/u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1/seg_fd/lexicon-7.ixix':
Invalid argument
> 20161211 182109 [] *    LUCY_FSFolder_Local_Open_FileHandle_IMP at core/Lucy/Store/FSFolder.c
line 118
> 20161211 182109 [] *    LUCY_Folder_Local_Open_In_IMP at core/Lucy/Store/Folder.c line
101
> 20161211 182109 [] *    LUCY_Folder_Open_In_IMP at core/Lucy/Store/Folder.c line 75
>
> There are two more failures they also failed due so similar reasons
>
> rename from '/u/smoke/presub/logs/cit-fg-adr-ndo-rtp.rajivg.1481473039.49384_cmode_1of1/.lucyindex/1/schema.temp'
to '/u/smoke/presub/logs/cit-fg-adr-ndo-rtp.rajivg.1481473039.49384_cmode_1of1/.lucyindex/1/schema_e4.json'
failed: No such file or directory
>
> Can't delete 'lexicon-3.ix'
>
> I believe all three are related to race condition while doing parallel indexing and should
go away with retries. However my retries started failing with different error which is strange
to me as if directory already exists shouldn't it skip from create attempt.
>
> 20161211 182109 [] * FAIL: [FAILED]: Retrying to add doc at path: /u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1
:  Couldn't create directory '/u/smoke/presub/logs/cit-fg-adr-neg-rtp.rajivg.1481473130.41339_cmode_1of1/.lucyindex/1':
No such file or directory
> 20161211 182109 [] *    LUCY_FSFolder_Initialize_IMP at core/Lucy/Store/FSFolder.c line
102
>
> So my all retry attempts were also failed.

These errors still look like multiple processes are modifying the index at the 
same time. Are you really sure that every indexer is created with an 
IndexManager and that every IndexManager is created with a `host` argument 
that is unique to each machine?

All these errors mean that there's something fundamentally wrong with your 
code or that you hit a bug in Lucy. The only type of error where it makes 
sense to retry is LockErr. All other errors are mostly fatal and could result 
in index corruption. Retrying will only mask an underlying problem in this case.

Unfortunately, I'm unable to help unless you provide some kind of 
self-contained, reproducible test case. I'm aware that this isn't easy, 
especially with multiple clients writing to a shared volume.

As I already hinted at, you might want to reconsider your architecture and use 
some kind of search server that uses an index on a local filesystem. There are 
ready-made platforms on top of Lucy like Dezi, but it isn't too hard to roll 
your own solution. This should result in better performance and makes behavior 
of your code more predictable.

Nick


Mime
View raw message