lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <mar...@rectangular.com>
Subject Re: [lucy-user] Indexing error message
Date Mon, 08 Dec 2014 23:18:16 GMT
On Mon, Dec 8, 2014 at 12:55 PM, Edwin Crockford <ecrockford@invicro.com> wrote:
> Hi Marvin,
>
> Thanks for the quick reply, here's a fragment of  the cfmeta.json file for
> the segment:
>
> {
>   "files": {
>     "documents.dat": {
>       "length": "17556716",
>       "offset": "0"
>     },
>     "documents.ix": {
>       "length": "238760",
>       "offset": "17556720"
>     },
>     "highlight.dat": {
>       "length": "47793",
>       "offset": "17795480"
>     },
>     "highlight.ix": {
>       "length": "0",
>       "offset": "17843280"
>     },
>
>
> Not quite sure what the format is but it  has a 0 length for "highlight.ix",
> highlight.dat has a largish length. Is this some failure in the highlighting
> mechansim?

Looking at the code in HighlightWriter.c, nothing jumps out at me.  I can't
see how it's possible to write to highlight.dat without also writing to
highlight.ix.  And for what it's worth, HighlightWriter's codebase has been
largely stable since 2009, recieving only minor modifications.

Disk filling up also seems unlikely -- lots of other files are written to at
the same time as highlight.ix (e.g. documents.ix) and those don't exhibit the
same problem; we should detect that a flush had failed when the file
descriptor gets closed; the "compound files" cf.dat and cfmeta.json get
written *after* highlight.ix, at which point you need *more* disk space.

Insead, I speculate what we are looking at is a different manifestation of the
same problem we talked about in August 2013.

    http://markmail.org/message/vynzixtoxfxhcx42

    > I believe we have traced the issue back to an interaction between two
    > different systems (one doing bulk updates and another doing on the fly
    > single document indexing) attempting updates at the same time. I
think there
    > was a way around the locking that caused the problem, does that seem
    > plausable?

    Yes, that makes sense. The error can be explained by having two Indexers
    trying to write to the same segment. One of them happens to delete the temp
    file "lextemp" first, and then the other can't find it and throws an
    exception.

    Only one Indexer may operate on a given index at a time. A BackgroundMerger
    may operate at the same time as an Indexer, but even it must
acquire the write
    lock briefly (once at the start of its run and once again at the end). While
    Lucy's locking APIs provide the technical capacity to disable the locking
    mechanism, it is not generally possible to get around the need for
locking in
    order to coordinate write access to an index.

Generally, when you disable locking and two writers attempt to write the same
segment, the first Indexer will crash before commit() completes and the index
will be left in a consistent state.  If you are unlucky, though, there's a
possibility you'll get corrupt data instead.

For that to happen, the second indexing process would have to start up while
the first was nearly done and in the process of assembling the compound file
`cf.dat` from temporary files such as `highlight.ix`.  The second process
"cleans up" the temp files from the "crashed" first process and initializes
new empty files.  The first process doesn't realize that its own
highlight.ix file has been clobbered and slurps the new empty file into
cf.dat.

How was your issue from last year resolved?

Marvin Humphrey

Mime
View raw message