lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gupta, Rajiv" <Rajiv.Gu...@netapp.com>
Subject RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c line 119
Date Fri, 09 Dec 2016 14:01:40 GMT
I'm getting this error very frequently now :(

BasicFlexGroup0_io_workload/pm_io/.lucyindex/1 :  input 47 too high 
S_fibonacci at core/Lucy/Index/IndexManager.c line 129

Is there any workaround? 

I'm using LightMergeManager I'm not sure if it is because of that. Should I stop that? 

Please help. Very frequently I'm getting it now.

-----Original Message-----
From: Gupta, Rajiv [mailto:Rajiv.Gupta@netapp.com] 
Sent: Thursday, December 08, 2016 4:00 PM
To: user@lucy.apache.org
Subject: RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c line 119

Fired multiple runs 10+ and all went through well except one where I got this error after
replacing default manager to 

manager =>  LightMergeManager->new( host => $self->{_hostname}."DEL"),

ERROR:
20161208 033346 [] * FAIL: FAILED AT initializing the IndexSearcher Couldn't get deletion
lock
20161208 033346 [] *    lucy_PolyReader_do_open at core/Lucy/Index/PolyReader.c line 344
20161208 033346 [] *    at /x/eng/bbrtp/users/rajivg/dotdev_052309_4015413_1612060522/test/nate/bin/../lib/NATE/LucyIndexerUtils.pm
line 3387.
20161208 033346 [] *    eval {...} called at /x/eng/bbrtp/users/rajivg/dotdev_052309_4015413_1612060522/test/nate/bin/../lib/NATE/LucyIndexerUtils.pm
line 3383
20161208 033346 [] *    eval {...} called at /x/eng/bbrtp/users/rajivg/dotdev_052309_4015413_1612060522/test/nate/bin/../lib/NATE/LucyIndexerUtils.pm
line 3381

All indexer operations except delete I wrapped around retries. So now I put retry against
this as well.

However, the search is now slower (after adding PolyReader/IndexReader). I used PolyReader
as in one of the forum it was mentioned that PolyReader has protection against some mem leak
issue. 

Any tips I can improve performance while using IndexReader?

Thanks much for all your support. 

Thanks,
Rajiv Gupta


-----Original Message-----
From: Gupta, Rajiv [mailto:Rajiv.Gupta@netapp.com] 
Sent: Wednesday, December 07, 2016 9:55 PM
To: user@lucy.apache.org
Subject: RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c line 119

However, since I'm searching and indexing the files from the same process and same system
should they need to be unique? Should I append something like <hostname>_search, <hostname>_index,
<hostname>_delete?

-Rajiv

-----Original Message-----
From: Gupta, Rajiv [mailto:Rajiv.Gupta@netapp.com] 
Sent: Wednesday, December 07, 2016 9:51 PM
To: user@lucy.apache.org
Subject: RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c line 119

I just now did that. 

For my doc regular update, search and delete I'm using LightMergemanager with my host name.
For adding end file marker I'm using regular manager with my host name added to conclude.
I have also put retries around almost all commits where I was getting errors. Small runs were
fine (there were any way fine) I have 5 large runs. 

I will update the results here. 

Thanks,
Rajiv Gupta

-----Original Message-----
From: Nick Wellnhofer [mailto:wellnhofer@aevum.de] 
Sent: Wednesday, December 07, 2016 9:38 PM
To: user@lucy.apache.org
Subject: Re: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c line 119

On 07/12/2016 13:23, Gupta, Rajiv wrote:
> * Indexer and log files are on NFS mount.

Have you read and understood Lucy::Docs::FileLocking? With NFS, you have to pass an IndexManager
object to every indexer and searcher.

     http://lucy.apache.org/docs/perl/Lucy/Docs/FileLocking.html

The fact that the index is on NFS probably also explains the performance problems you reported
earlier.

Nick


Mime
View raw message