hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki ...@getopt.org>
Subject Lock manager
Date Tue, 14 Feb 2006 09:55:55 GMT

I noticed that there are some additions being made to the 
JobSubmissionProtocol. While we're on this subject, I'd like to propose 
the following addition.

Currently there is no mechanism to coordinate concurrent access to 
specific resources across the cluster. This is apparent in Nutch in the 
protocol plugins, which need this "global lock" mechanism to control 
simultaneous access to remote hosts (at the moment the workaround is to 
execute just one task simultaneously to handle the locking inside a 
single JVM).

We could devise a mechanism based on files, residing on DFS, but it 
would be too heavyweight - in the above case it would mean 
creating/reading/deleting millions of files per job.

I propose to add a simple locking mechanism to JobTracker, which would 
hold locks specific to a given job. Then each task executing on a 
tasktracker could send requests  to JobTracker to acquire/release such 
locks (identified by strings, perhaps with some grouping mechanism), 
with the usual semantics. The cleanup in case of crashed jobs would be 
simple - whenever a job is finished, JobTracker would remove all locks 
related to that job.

In this schema it would be the JobTracker node that runs this lock 
manager, but I can imagine that we could designate any of the 
tasktracker nodes to run this "lock manager task" for each job, if it 
ever becomes a scalability problem. We could also go even further and 
create two types of lock managers - global lock manager (for resources 
unique per cluster) and per-job lock managers (for resources unique per 

Any comments?

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

View raw message