Mailing-List: contact lucy-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of gorankent@gmail.com
 designates 209.85.212.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <20111118151820.GA19347@rectangular.com>
References: 
 <CAKn6EF-o+aNM5icZg=7O2+7uVfSce_3xbMC7=Ym+jxnE4R9juw@mail.gmail.com>
	<20111117061703.GA27162@rectangular.com>
	<CAKn6EF-e_Vhzv9opkW+EysMh7Et73RS-1rEOgvvE-gmOom1cMQ@mail.gmail.com>
	<20111117070813.GA27239@rectangular.com>
	<CAKn6EF-9UEX9d3OoBSBmP9r6NyQd=Ag47YkEPTd21X-Lq_Pe5w@mail.gmail.com>
	<20111118151820.GA19347@rectangular.com>
Date: Fri, 18 Nov 2011 20:14:20 +0200
Message-ID: 
 <CAKn6EF_hE0VX8fdqWGnm4=zOmDd9kQsAjGioyTxf1JZNxN-7hg@mail.gmail.com>
From: goran kent <gorankent@gmail.com>
To: lucy-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: Re: [lucy-user] Couldn't completely remove 'seg_N'

On Fri, Nov 18, 2011 at 5:18 PM, Marvin Humphrey <marvin@rectangular.com> w=
rote:
>> The lockfile contains:
>> {
>> =A0 "host": "host6",
>> =A0 "name": "write",
>> =A0 "pid": "24342"
>> }
>
> OK, all that looks correct. =A0Also, since the lockfile is still there an=
d
> definitely corresponds to the process that crashed, we can assume that no
> other process has messed with the index directory since.
>
> Question: is there a seg_2 folder in the index dir? =A0If so, is there an=
ything
> inside it?

Yes, seg_2 was there - it also had the same timestamp as the lockfile,
implying it had been created by the same process that created the
lockfile itself.
Secondly, no it was empty.

> It could be NFS cache consistency: a deletion operation succeeds, and the=
 item
> is really gone from the NFS drive, but the local cache of the NFS client
> doesn't get updated in time and a subsequent check on whether the item ex=
ists
> returns an incorrect result.
>
> =A0 =A0http://nfs.sourceforge.net/#faq_a8
>
> =A0 =A0Perfect cache coherency among disparate NFS clients is very expens=
ive to
> =A0 =A0achieve, so NFS settles for something weaker that satisfies the
> =A0 =A0requirements of most everyday types of file sharing.
>
> A tremendous amount of energy has gone into making NFS mimic local file s=
ystem
> behaviors as closely as possible, both by the NFS devs and by us (see
> <http://incubator.apache.org/lucy/docs/perl/Lucy/Docs/FileLocking.html>) =
but
> it's a very hard problem and compromises are impossible to avoid.

Some food for thought, thanks.  I'll start looking into my index store
servers and their NFS exports.

> Best practice would be to avoid writing to Lucy indexes on NFS drives if
> possible. =A0Read performance is going to be lousy anyway unless you make=
 the
> NFS mount read-only.

Just too much data (and we needed redundancy), and the load needs to
be spread across as many storage nodes as possible (we have separate
source-store and index-store servers).  When all the indexing machines
are grinding away in unison, sucking from the source-store servers
(via NFS), and writing to the index-store servers (also via NFS), the
load can get quite high and some NFS tweaking has been done on the
source/target servers.

Merging nodes reads indexes from the index-store servers (NFS) and
writes them on the search-server (NFS) nodes themselves.  Searching is
then always local.

The load at the moment is negligible, so I know it's not causing a
problem with NFS - however, with NFS you never know, so I'll be
focusing on that next.

Thanks for your comments!