lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From karl wettin <ka...@snigel.dnsalias.net>
Subject Re: N-gram layer
Date Fri, 12 Mar 2004 01:54:22 GMT
On Sun, 1 Feb 2004 13:12:32 -0800 (PST)
Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:

> Looking forward to the contribution.

Sorry for the delay, but I've had quite some workload lately, and then I
moved between apartments. I'm back and I'm ready to spend some time.

I gave up detecting the language of a query. It is very possbile indeed
and I got great results with Weka, but takes too much time: 5-50 seconds
on my Pentium M. 

However, I'm still working on the "autoanalytic stemmer", atleast in my
head. I've started to feed my index with docuemnts tagged with the
language in a field, and thought it should analyze (still the n-gram
approach) all  words of a specific language to find stemming rules for
each and every language. The output can be used per language stemming,
BUT hopefully I'll be able to use this data to create my generic
stemmer.

The language models and inflectional form extraction should be based on
the index content, but I can't seem to find out how to access the terms
of a specific set of documents. Of course, I could just query my index
and start working on the data, building my own trie-pattern, but I'm 
sure I don't have to.

I've been browsing the list archives and API for several days without
finding out how to iterate the (distinct/unique) terms of the index
or a specific set of documents. 

How do I do that? 



-- 

karl

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message