lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Socket and file locks
Date Mon, 23 Nov 2009 18:31:16 GMT
On Sun, Nov 22, 2009 at 10:36:57AM +0000, Thomas Mueller (JIRA) wrote:

> Thomas Mueller commented on LUCENE-1877:
> ----------------------------------------
> > take it somewhere other than this closed issue.
> Yes, where?

The java-dev list.

> > shouldn't active code like that live in the application layer?
> Why?

You can all but guarantee that polling will work at the app layer, because you
can have almost full control over process priority.
If the polling code is lower down and hidden away, then it worries me that a
lock might be swept away by another process, and by the time the original
process realizes that it doesn't hold the lock anymore, the damage could
already have been done.  Unless I'm missing something, it doesn't seem like a
failsafe design.

But this is "theoretical", I suppose:

> I'm just trying to say that in theory, the thread is problematic, but in
> practice it isn't. While file locking is not a problem in theory, but in
> practice.

Heh. :)

> > What happens when the app sleeps?
> Good question! Standby / hibernate are not supported. I didn't think about
> that. Is there a way to detect the wakeup?

Not sure.  

FYI, I'm only an indirect contributor to Java Lucene.  My main projects are
Lucy and KinoSearch, loose ports to C.  I know the problem domain intimately,
but my Java skills are sketchy.

> > host name and the pid
> Yes. It is not so easy to get the PID in Java, I found:
> "ManagementFactory.getRuntimeMXBean().getName()". 

A web search for "java process id" turns up a bazillion hits about how to hack
up a PID.

How annoying.  This seems to me like a case of the perfect being the enemy of
the good.  How many machines that run Java are running operating systems that
have no support for PIDs?

Hasn't somebody open sourced a "GiveMeTheFrikkinPID" library yet?

> What do you do if the lock was generated by another machine? 

Require that all machines participating in the writer pool supply a unique
host ID as part of the locking API.  Store that host ID in the lockfile and
only allow machines to sweep stale files that they own.

Unfortunately, that's not failsafe either, though: misconfiguration leads to
index corruption rather than deadlock, when two machines that use identical
host IDs sweep each others lockfiles and write simultaneously.

> I tried with using a server socket, so you need the IP address, but
> unfortunately, sometimes the network is not configured correctly (but maybe
> it's possible to detect that). Maybe the two machines can't access each
> other over TCP/IP.

This is an intriguing approach.  Can it be designed to be failsafe?

If the server and the client can't access each other, that's failsafe at
least, because the client will simply fail to acquire the lock.

But if a client is misconfigured, could it contact the wrong host,
successfully open a port that coincidentally happens to be open, believe it
has acquired the lock and corrupt the index?  If so, could some sort of
handshake prevent that?

I'm also curious if we can use this approach for read locking.  For that, you
need a reference counting scheme -- one ref for each reader accessing the
index.  Is that possible under the socket model?

> > hard links
> Yes, but it looks like this doesn't work always.

It is theoretically possible for the link() call to return false incorrectly
when the hard link has actually been created, for instance because a network
problem prevents the "success" packet from getting back to the client from the

However, this is failsafe, because the requestor will not believe that the
lock has been secured and thus won't write.  That process won't be able to
sweep away the orphaned lock file itself, but once it exits, a graceful
recovery will occur.

Marvin Humphrey

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message