Mailing-List: contact user-help@lucy.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@lucy.apache.org
From: "Gupta, Rajiv" <Rajiv.Gupta@netapp.com>
To: "user@lucy.apache.org" <user@lucy.apache.org>
Thread-Topic: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c
 line 119
Thread-Index: AdJPkWW3M42uhmZBTuSaRV1BWQdqXgASsAfQACfG9QAAAD47UAAJ+V+AAABDUoAAAEeFQAAlmR3AADnzPrA=
Date: Fri, 9 Dec 2016 14:01:40 +0000
Message-ID: <CY1PR06MB217243235E91EA8613C7297694870@CY1PR06MB2172.namprd06.prod.outlook.com>
References: <CY1PR06MB2172D9FFA47BAFD82A51351794820@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB21725B216E89D7369DA6F0AA94820@CY1PR06MB2172.namprd06.prod.outlook.com>
 <cd2168fb-227b-d672-9359-1a32e29869a7@aevum.de>
 <CY1PR06MB21723A402467EA7A1E3ACD2294850@CY1PR06MB2172.namprd06.prod.outlook.com>
 <1fda7218-5576-89da-03b4-6ffe7155462b@aevum.de>
 <CY1PR06MB21724B8E85F44099F82B64CF94850@CY1PR06MB2172.namprd06.prod.outlook.com>
 <CY1PR06MB21724E5BD5F52E852B3CFBBE94850@CY1PR06MB2172.namprd06.prod.outlook.com>
 <BY2PR06MB2167AC743BB46B64BBFACFAA94840@BY2PR06MB2167.namprd06.prod.outlook.com>
In-Reply-To: <BY2PR06MB2167AC743BB46B64BBFACFAA94840@BY2PR06MB2167.namprd06.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: netapp.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Dec 2016 14:01:40.3242
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR06MB2169
X-OriginatorOrg: netapp.com
Subject: RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder.c
 line 119
archived-at: Fri, 09 Dec 2016 14:02:21 -0000

I'm getting this error very frequently now :(

BasicFlexGroup0_io_workload/pm_io/.lucyindex/1 :  input 47 too high=20
S_fibonacci at core/Lucy/Index/IndexManager.c line 129

Is there any workaround?=20

I'm using LightMergeManager I'm not sure if it is because of that. Should I=
 stop that?=20

Please help. Very frequently I'm getting it now.

-----Original Message-----
From: Gupta, Rajiv [mailto:Rajiv.Gupta@netapp.com]=20
Sent: Thursday, December 08, 2016 4:00 PM
To: user@lucy.apache.org
Subject: RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder=
.c line 119

Fired multiple runs 10+ and all went through well except one where I got th=
is error after replacing default manager to=20

manager =3D>  LightMergeManager->new( host =3D> $self->{_hostname}."DEL"),

ERROR:
20161208 033346 [] * FAIL: FAILED AT initializing the IndexSearcher Couldn'=
t get deletion lock
20161208 033346 [] *    lucy_PolyReader_do_open at core/Lucy/Index/PolyRead=
er.c line 344
20161208 033346 [] *    at /x/eng/bbrtp/users/rajivg/dotdev_052309_4015413_=
1612060522/test/nate/bin/../lib/NATE/LucyIndexerUtils.pm line 3387.
20161208 033346 [] *    eval {...} called at /x/eng/bbrtp/users/rajivg/dotd=
ev_052309_4015413_1612060522/test/nate/bin/../lib/NATE/LucyIndexerUtils.pm =
line 3383
20161208 033346 [] *    eval {...} called at /x/eng/bbrtp/users/rajivg/dotd=
ev_052309_4015413_1612060522/test/nate/bin/../lib/NATE/LucyIndexerUtils.pm =
line 3381

All indexer operations except delete I wrapped around retries. So now I put=
 retry against this as well.

However, the search is now slower (after adding PolyReader/IndexReader). I =
used PolyReader as in one of the forum it was mentioned that PolyReader has=
 protection against some mem leak issue.=20

Any tips I can improve performance while using IndexReader?

Thanks much for all your support.=20

Thanks,
Rajiv Gupta


-----Original Message-----
From: Gupta, Rajiv [mailto:Rajiv.Gupta@netapp.com]=20
Sent: Wednesday, December 07, 2016 9:55 PM
To: user@lucy.apache.org
Subject: RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder=
.c line 119

However, since I'm searching and indexing the files from the same process a=
nd same system should they need to be unique? Should I append something lik=
e <hostname>_search, <hostname>_index, <hostname>_delete?

-Rajiv

-----Original Message-----
From: Gupta, Rajiv [mailto:Rajiv.Gupta@netapp.com]=20
Sent: Wednesday, December 07, 2016 9:51 PM
To: user@lucy.apache.org
Subject: RE: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder=
.c line 119

I just now did that.=20

For my doc regular update, search and delete I'm using LightMergemanager wi=
th my host name. For adding end file marker I'm using regular manager with =
my host name added to conclude. I have also put retries around almost all c=
ommits where I was getting errors. Small runs were fine (there were any way=
 fine) I have 5 large runs.=20

I will update the results here.=20

Thanks,
Rajiv Gupta

-----Original Message-----
From: Nick Wellnhofer [mailto:wellnhofer@aevum.de]=20
Sent: Wednesday, December 07, 2016 9:38 PM
To: user@lucy.apache.org
Subject: Re: [lucy-user] LUCY_Folder_Open_Out_IMP at core/Lucy/Store/Folder=
.c line 119

On 07/12/2016 13:23, Gupta, Rajiv wrote:
> * Indexer and log files are on NFS mount.

Have you read and understood Lucy::Docs::FileLocking? With NFS, you have to=
 pass an IndexManager object to every indexer and searcher.

     http://lucy.apache.org/docs/perl/Lucy/Docs/FileLocking.html

The fact that the index is on NFS probably also explains the performance pr=
oblems you reported earlier.

Nick