lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luca Rondanini <luca.rondan...@gmail.com>
Subject Language Identifier with Lucene?
Date Sat, 22 Oct 2011 00:49:55 GMT
Hi all,

I usually use Nutch for this but, just for fun, I tried to create a language
identifier based on Lucene only.

I had a really small set of "training data": 10 files (roughly 2M each) for
10 languages. I indexed those files using an NGram analyzer.

I have to say that I was not expecting much...but the results seem amazing!

What do you think of this approach?

Beside the good results, that could easily be false positives due to the
small training set, I wonder if maybe I am misunderstanding Lucene's VSM!

I'll try to increase the "training" data and see what happen changing NGram
size too! I'll keep you posted, meanwhile let me know what you think.

Thanks,
Luca

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message