hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DES <sant....@gmail.com>
Subject Hadoop and Lucene write lock
Date Thu, 26 Jul 2007 23:06:55 GMT
hello,

I tried nutch with hadoop nightly builds (in hudson #135 and newer) and got
following problem:


java.io.IOException: Lock obtain timed out:
Lock@hdfs://xxx.xxx.xxx.xxx:9000/user/nutch/crawl/indexes/part-00020/write.lock
	at org.apache.lucene.store.Lock.obtain(Lock.java:69)
	at org.apache.lucene.index.IndexReader.aquireWriteLock(IndexReader.java:526)

	at org.apache.lucene.index.IndexReader.deleteDocument(IndexReader.java:551)
	at org.apache.nutch.indexer.DeleteDuplicates.reduce(DeleteDuplicates.java:451)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java
:323)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1763)


I think the reason could be the lucene locks.
I just tried following code and got exactly the same error:

String indexPath="crawl/index";
Path index=new Path(indexPath);
Configuration conf = NutchConfiguration.create();
JobConf job = new NutchJob(conf);
FileSystem fs = FileSystem.get(job);
FsDirectory dir=new FsDirectory(fs, index, false, conf);
IndexReader reader = IndexReader.open(dir);
reader.deleteDocument(0);

can somebody tell me if there is a solution for that? or should I just drop
back to older hadoop version? (e.g. 0.12.x)

thanks

des

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message