directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <>
Subject Re: Problem with indexes...
Date Fri, 16 May 2014 08:47:09 GMT
Le 07/05/2014 18:17, mark debusschere a écrit :
> Was this issue related to M16 or has it been outstanding?  I have had countless
> problems with indexes, a few self induced, but recently indexes just stopped working.
> This was after server failed to start due  to system schema already present, then
> second attempt server started but app failed to function as all searches were failing.
(first, sorry for the delayed answer, your mail just made it to my inbox
due to heavy issues in The ASF mailers which has stopped working for 5

There is not one single response to your question. Let me try to give
you some insight.

There are many different use cases, and potantial kind of failures :
- indexes you create before starting the server and injecting data
- indexes you create after having injected data into the server
- we have user indexes (those you create yourself) and system indexes
(those used internally by the server)

The bugs we had was related to the missing initialization of indexes
created after some data were injected. This has been fixed partially in
M16 and in trunk.

Beside this obvious bug, there is a way more critical issue we are
working on for months now : the backend we are using (JDBM) is ok when
it comes to deal with one single BTree (an index is just a BTree, well,
a bit more), but it's not supporting transactions across btrees.

What does it means, and what are the consequences ? Pretty heavy :

- when some user is updating an entry, it will impact tens of indexes
- if at the same time another user is searching the server, it's likely
to use the very same indexes
- *if* this occurs at the very same time, then the updating client might
modify an idex which is used by one of the reader, leading to some
inconsistancies, and an error.

We have added many locks to try to workaround this issue, we modified
the way we fetched the candidates to limit the time it takes to build
this set of candidates - and as a consequence the time during which we
are holding an index -. We     also added a journal in the backend to be
able to recover when a crash occurs. All those modifications have
improved the  situation, up to a point the server now don't crash when
there are some concurrent reads and writes, but it comes to a price :
all the operations are way longer than they could be.

We are now working on a complete replacement for the JDBM backend :
Mavibot. It will allow us to process an update as a whole, ie all the
indexes will be updated in one shot, or not at all. The searches will be
guaranteed to work as they will *always* proceed on an existing
revision. That will take a bit of time to achieve this new backend, but
we are heavily working on it.

I hope I clarified some aspects of the probelm, and the solution we are
working on...


View raw message